Nov 03 21:27:03.485269 ip-10-0-131-253 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 21:27:03.485278 ip-10-0-131-253 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 21:27:03.485285 ip-10-0-131-253 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 21:27:03.485531 ip-10-0-131-253 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 21:27:13.647091 ip-10-0-131-253 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 21:27:13.647111 ip-10-0-131-253 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1aa59ee141614e239e03c91e4c35c583 -- Nov 03 21:29:02.983942 ip-10-0-131-253 systemd[1]: Starting Kubernetes Kubelet... Nov 03 21:29:03.432295 ip-10-0-131-253 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:29:03.432295 ip-10-0-131-253 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 21:29:03.432295 ip-10-0-131-253 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:29:03.432295 ip-10-0-131-253 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:29:03.444892 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444797 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:29:03.444892 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444886 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:29:03.444892 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444891 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:29:03.444892 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444896 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:29:03.444892 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444900 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444904 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444908 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444912 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444916 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444925 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444929 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444932 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444936 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444940 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444944 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444948 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444952 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444956 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444960 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444964 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444968 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444973 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444981 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444985 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:29:03.445187 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444991 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444995 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.444999 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445003 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445007 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445011 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445015 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445019 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445023 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445028 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445032 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445042 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445046 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445050 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445055 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445060 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445064 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445069 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445073 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:29:03.445841 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445078 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445082 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445088 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445092 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445101 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445105 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445113 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445119 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445123 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445128 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445132 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445137 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445141 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445145 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445151 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445155 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445160 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445169 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445173 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:29:03.446284 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445178 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445183 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445187 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445191 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445195 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445199 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445204 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445208 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445215 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445222 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445233 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445238 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445245 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445249 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445254 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445258 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445262 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445267 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445271 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:29:03.446847 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445275 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445280 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445284 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445293 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445297 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445302 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445308 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445312 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445317 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445322 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445327 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445331 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445336 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445341 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445346 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445350 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445359 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445363 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445367 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445371 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:29:03.447599 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445376 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:29:03.448401 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445384 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:29:03.448401 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445388 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:29:03.448401 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445393 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:29:03.448401 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.445398 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:29:03.448641 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448624 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:29:03.448641 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448643 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448649 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448654 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448672 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448677 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448681 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448686 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448690 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448695 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448704 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448708 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448713 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448717 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448721 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448726 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448730 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448739 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448746 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448752 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:29:03.448758 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448757 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448763 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448772 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448777 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448781 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448785 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448790 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448795 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448799 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448803 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448808 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448812 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448816 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448821 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448825 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448834 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448838 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448842 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448847 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:29:03.449673 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448851 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448856 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448860 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448864 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448868 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448872 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448877 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448882 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448892 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448898 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448903 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448910 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448915 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448920 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448924 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448929 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448934 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448938 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448942 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:29:03.450165 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448946 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448955 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448960 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448964 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448968 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448973 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448977 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448982 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448986 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448991 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.448996 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449001 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449005 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449010 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449019 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449023 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449027 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449031 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449035 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449039 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:29:03.450603 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449043 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449047 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449051 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449055 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449059 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449065 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449073 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449078 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449082 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449087 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449091 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449096 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449100 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449105 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449108 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449112 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449117 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449121 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449125 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449134 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:29:03.451098 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449140 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449145 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449149 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449153 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449161 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449166 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449170 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.449174 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450308 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450321 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450324 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450329 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450334 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450338 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450341 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450344 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450347 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450350 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450353 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:29:03.451553 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450355 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450358 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450360 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450363 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450366 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450368 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450371 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450373 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450376 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450378 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450381 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450383 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450386 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450388 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450392 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450395 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450397 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450400 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450402 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450406 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:29:03.452004 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450409 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450412 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450415 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450417 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450419 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450422 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450424 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450427 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450429 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450432 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450434 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450436 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450439 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450441 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450444 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450446 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450449 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450451 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450455 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:29:03.452466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450458 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450460 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450463 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450466 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450468 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450471 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450474 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450477 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450479 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450482 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450484 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450487 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450490 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450492 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450495 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450498 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450500 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450503 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450506 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450509 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:29:03.452920 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450511 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450513 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450516 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450518 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450520 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450523 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450525 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450527 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450530 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450532 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450535 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450537 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450539 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450542 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450545 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450547 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450550 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450552 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450555 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:29:03.453384 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450559 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450561 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450564 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450566 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450568 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450571 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450573 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450576 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450578 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450580 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450583 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450586 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450589 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450591 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450594 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450597 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.450599 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450716 2584 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450724 2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450731 2584 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450735 2584 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 21:29:03.453861 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450740 2584 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450743 2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450747 2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450751 2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450754 2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450758 2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450761 2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450765 2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450768 2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450771 2584 flags.go:64] FLAG: --cgroup-root="" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450774 2584 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450777 2584 flags.go:64] FLAG: --client-ca-file="" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450779 2584 flags.go:64] FLAG: --cloud-provider="external" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450782 2584 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450787 2584 flags.go:64] FLAG: --cluster-domain="" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450790 2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450793 2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450796 2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450800 2584 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450804 2584 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450807 2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450810 2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450813 2584 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450816 2584 flags.go:64] FLAG: --contention-profiling="false" Nov 03 21:29:03.454337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450820 2584 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450823 2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450826 2584 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450829 2584 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450833 2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450836 2584 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450840 2584 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450842 2584 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450845 2584 flags.go:64] FLAG: --enable-server="true" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450848 2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450853 2584 flags.go:64] FLAG: --event-burst="100" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450856 2584 flags.go:64] FLAG: --event-qps="50" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450859 2584 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450862 2584 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450865 2584 flags.go:64] FLAG: --eviction-hard="" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450870 2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450872 2584 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450876 2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450879 2584 flags.go:64] FLAG: --eviction-soft="" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450882 2584 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450885 2584 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450887 2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450890 2584 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450893 2584 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450896 2584 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 21:29:03.454929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450898 2584 flags.go:64] FLAG: --feature-gates="" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450905 2584 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450909 2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450912 2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450915 2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450918 2584 flags.go:64] FLAG: --healthz-port="10248" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450921 2584 flags.go:64] FLAG: --help="false" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450923 2584 flags.go:64] FLAG: --hostname-override="ip-10-0-131-253.ec2.internal" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450927 2584 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450930 2584 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450933 2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450936 2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450940 2584 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450943 2584 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450946 2584 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450949 2584 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450951 2584 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450954 2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450957 2584 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450960 2584 flags.go:64] FLAG: --kube-reserved="" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450963 2584 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450966 2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450969 2584 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450972 2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 21:29:03.455502 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450975 2584 flags.go:64] FLAG: --lock-file="" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450978 2584 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450981 2584 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450984 2584 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450989 2584 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450992 2584 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450995 2584 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.450998 2584 flags.go:64] FLAG: --logging-format="text" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451001 2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451006 2584 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451009 2584 flags.go:64] FLAG: --manifest-url="" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451012 2584 flags.go:64] FLAG: --manifest-url-header="" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451017 2584 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451020 2584 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451023 2584 flags.go:64] FLAG: --max-pods="110" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451027 2584 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451030 2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451034 2584 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451037 2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451040 2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451043 2584 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451046 2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451054 2584 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451057 2584 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451060 2584 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 21:29:03.456086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451063 2584 flags.go:64] FLAG: --pod-cidr="" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451066 2584 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451070 2584 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451073 2584 flags.go:64] FLAG: --pods-per-core="0" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451076 2584 flags.go:64] FLAG: --port="10250" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451078 2584 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451082 2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-091b66530eba67f85" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451085 2584 flags.go:64] FLAG: --qos-reserved="" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451088 2584 flags.go:64] FLAG: --read-only-port="10255" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451091 2584 flags.go:64] FLAG: --register-node="true" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451094 2584 flags.go:64] FLAG: --register-with-taints="" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451099 2584 flags.go:64] FLAG: --registry-burst="10" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451101 2584 flags.go:64] FLAG: --registry-qps="5" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451104 2584 flags.go:64] FLAG: --reserved-cpus="" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451107 2584 flags.go:64] FLAG: --reserved-memory="" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451111 2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451116 2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451119 2584 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451122 2584 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451125 2584 flags.go:64] FLAG: --runonce="false" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451127 2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451130 2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451134 2584 flags.go:64] FLAG: --seccomp-default="false" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451136 2584 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451139 2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451143 2584 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 21:29:03.456677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451146 2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451149 2584 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451152 2584 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451155 2584 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451158 2584 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451161 2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451164 2584 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451167 2584 flags.go:64] FLAG: --system-cgroups="" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451172 2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451177 2584 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451179 2584 flags.go:64] FLAG: --tls-cert-file="" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451182 2584 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451187 2584 flags.go:64] FLAG: --tls-min-version="" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451190 2584 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451192 2584 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451195 2584 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451198 2584 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451201 2584 flags.go:64] FLAG: --v="2" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451205 2584 flags.go:64] FLAG: --version="false" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451209 2584 flags.go:64] FLAG: --vmodule="" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451214 2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451217 2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451367 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451371 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:29:03.457418 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451375 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451377 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451380 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451383 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451386 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451389 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451391 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451394 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451397 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451400 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451403 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451405 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451408 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451410 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451413 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451415 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451419 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451421 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451424 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451426 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:29:03.458409 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451429 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451432 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451435 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451437 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451440 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451443 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451445 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451447 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451450 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451453 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451458 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451461 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451464 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451467 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451470 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451473 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451475 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451478 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451480 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:29:03.459232 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451482 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451485 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451488 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451490 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451493 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451495 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451498 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451502 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451505 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451509 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451512 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451514 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451517 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451520 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451522 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451524 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451527 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451530 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451532 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451535 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:29:03.459992 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451538 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451540 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451543 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451546 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451549 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451551 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451554 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451557 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451560 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451562 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451565 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451567 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451570 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451573 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451575 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451579 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451581 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451584 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451586 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:29:03.460466 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451589 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451591 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451595 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451598 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451601 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451603 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451606 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451608 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451611 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451613 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451616 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451618 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451620 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451623 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451626 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451628 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451632 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451634 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451637 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451639 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:29:03.461021 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451642 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451644 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451647 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451649 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451652 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451654 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.451691 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451728 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451732 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451736 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451739 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451741 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451744 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451747 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451749 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451754 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451756 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:29:03.461879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451759 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451762 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451764 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451767 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451769 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451772 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451775 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451778 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451780 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451783 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451786 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451789 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451792 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451794 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451797 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451799 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451802 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451805 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451807 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:29:03.462585 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451810 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451812 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451815 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451817 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451820 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451822 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451825 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451828 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451831 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451833 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451836 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451840 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451843 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451846 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451848 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451850 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451853 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451855 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451858 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:29:03.463185 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451861 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451865 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451869 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451871 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451875 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451879 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451882 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451884 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451887 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451890 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451892 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451895 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451897 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451900 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451902 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451905 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451908 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451911 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451915 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:29:03.463908 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451919 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451922 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451925 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451927 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451930 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451934 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451937 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451939 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451942 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451945 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451947 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451949 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451952 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451954 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451957 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451959 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451962 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451964 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451968 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451970 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:29:03.464681 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451973 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451975 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451977 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451980 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451982 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451985 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451987 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451990 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451992 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451995 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.451997 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.452000 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.452002 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.452005 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.452009 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.452012 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.452014 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.452018 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.452020 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.459758 2584 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 21:29:03.465257 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.459777 2584 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.460839 2584 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.460852 2584 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.460936 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.460947 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.460954 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.460959 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.460964 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.460968 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.460972 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.460977 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.460981 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.460985 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.460989 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.460994 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.460998 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461002 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461006 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:29:03.465906 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461011 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461015 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461019 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461023 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461027 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461032 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461038 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461042 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461046 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461051 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461055 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461059 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461064 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461068 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461075 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461079 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461084 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461088 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461092 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461096 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:29:03.466310 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461100 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461105 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461109 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461113 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461117 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461121 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461125 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461129 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461133 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461137 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461142 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461146 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461151 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461155 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461159 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461163 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461167 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461172 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461175 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461180 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:29:03.466830 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461184 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461188 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461193 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461197 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461201 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461205 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461210 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461215 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461220 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461224 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461228 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461232 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461236 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461240 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461245 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461249 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461253 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461257 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461262 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:29:03.467356 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461266 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461270 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461274 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461278 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461283 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461286 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461291 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461295 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461298 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461303 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461308 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461312 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461316 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461320 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461324 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461329 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461333 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461337 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461341 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461346 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:29:03.467903 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461350 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:29:03.468461 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461359 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:29:03.468461 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461364 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:29:03.468461 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461368 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:29:03.468461 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461372 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:29:03.468461 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461377 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:29:03.468461 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461381 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:29:03.468461 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461386 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:29:03.468461 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461390 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:29:03.468461 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461394 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:29:03.468461 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461399 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:29:03.468461 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461403 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:29:03.468461 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.461442 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:29:03.468461 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461680 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:29:03.468461 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461689 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:29:03.468461 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461694 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:29:03.468461 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461698 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461703 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461707 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461711 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461715 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461719 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461724 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461728 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461732 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461736 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461743 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461750 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461754 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461760 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461764 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461770 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461774 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461778 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461784 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:29:03.469142 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461788 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461792 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461796 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461800 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461805 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461809 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461813 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461817 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461821 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461826 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461829 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461834 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461838 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461842 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461846 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461850 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461854 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461859 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461863 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:29:03.469991 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461867 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461871 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461875 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461879 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461883 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461887 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461890 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461894 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461898 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461903 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461907 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461912 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461916 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461921 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461926 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461930 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461936 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461942 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461946 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461951 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:29:03.470776 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461956 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461960 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461965 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461969 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461973 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461978 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461982 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461986 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461991 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461995 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.461999 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462004 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462008 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462012 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462016 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462020 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462024 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462028 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462033 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462037 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:29:03.471472 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462041 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462045 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462050 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462054 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462058 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462062 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462067 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462071 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462075 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462079 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462083 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462087 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462091 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462094 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462098 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462102 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462106 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462110 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462114 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462119 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:29:03.472001 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462123 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:29:03.472492 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462127 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:29:03.472492 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462132 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:29:03.472492 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462136 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:29:03.472492 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462140 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:29:03.472492 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.462179 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:29:03.472492 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462232 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:29:03.472492 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462238 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:29:03.472492 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462245 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:29:03.472492 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462250 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:29:03.472492 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462255 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:29:03.472492 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462261 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:29:03.472492 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462266 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:29:03.472492 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462270 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:29:03.472492 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462274 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:29:03.472492 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462279 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:29:03.472492 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462283 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462288 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462292 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462296 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462301 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462305 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462309 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462314 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462318 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462322 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462326 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462330 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462334 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462338 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462342 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462346 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462350 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462354 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462359 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:29:03.472878 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462363 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462367 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462371 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462375 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462379 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462388 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462393 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462397 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462401 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462405 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462409 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462413 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462418 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462422 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462426 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462430 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462436 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462441 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462445 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462450 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:29:03.473371 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462454 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462458 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462462 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462466 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462471 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462475 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462479 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462483 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462487 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462492 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462496 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462500 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462504 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462508 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462512 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462519 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462525 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462532 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462537 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:29:03.473879 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462541 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462546 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462550 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462554 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462559 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462563 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462567 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462571 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462575 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462579 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462583 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462588 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462592 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462596 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462599 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462603 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462608 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462611 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462615 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462619 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:29:03.474372 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462624 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462628 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462632 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462638 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462642 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462646 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462651 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462655 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462673 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462679 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462685 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462690 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462696 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462700 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462704 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462708 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462713 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:03.462717 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.462795 2584 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 21:29:03.474840 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.465452 2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 21:29:03.475380 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.466507 2584 server.go:1014] "Starting client certificate rotation" Nov 03 21:29:03.475380 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.466603 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 21:29:03.475380 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.466635 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 21:29:03.475380 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.469183 2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 21:29:03.475380 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.471304 2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 21:29:03.483103 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.483090 2584 log.go:25] "Validated CRI v1 runtime API" Nov 03 21:29:03.486342 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.486328 2584 log.go:25] "Validated CRI v1 image API" Nov 03 21:29:03.488680 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.488639 2584 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 21:29:03.491334 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.491316 2584 fs.go:135] Filesystem UUIDs: map[08600e6d-1f1b-4b84-aa43-a329ae1bb807:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 db89b5c9-e9d8-4271-a861-49f1a1341575:/dev/nvme0n1p3] Nov 03 21:29:03.491386 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.491334 2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 21:29:03.496622 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.496493 2584 manager.go:217] Machine: {Timestamp:2026-11-03 21:29:03.494941206 +0000 UTC m=+0.387048673 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199888 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cdf649b406393e188eb5da64f21d7 SystemUUID:ec2cdf64-9b40-6393-e188-eb5da64f21d7 BootID:1aa59ee1-4161-4e23-9e03-c91e4c35c583 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c7:7d:b6:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c7:7d:b6:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:6a:db:da:95:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 21:29:03.496622 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.496617 2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 21:29:03.496748 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.496713 2584 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 21:29:03.499405 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.499380 2584 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 21:29:03.499531 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.499408 2584 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-253.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 21:29:03.499571 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.499545 2584 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 21:29:03.499571 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.499553 2584 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 21:29:03.499622 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.499588 2584 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 21:29:03.499622 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.499607 2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 21:29:03.500621 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.500612 2584 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 21:29:03.503732 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.503721 2584 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 21:29:03.503842 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.503833 2584 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 21:29:03.504496 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.504487 2584 kubelet.go:493] "Attempting to sync node with API server" Nov 03 21:29:03.504528 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.504500 2584 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 21:29:03.504528 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.504510 2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 21:29:03.504528 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.504519 2584 kubelet.go:405] "Adding apiserver pod source" Nov 03 21:29:03.504528 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.504528 2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 21:29:03.504933 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.504919 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 21:29:03.507559 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.507542 2584 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 21:29:03.513392 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.513377 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 21:29:03.514338 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.514322 2584 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 21:29:03.514380 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.514373 2584 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 21:29:03.514586 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.514574 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 21:29:03.514625 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.514595 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 21:29:03.514625 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.514605 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 21:29:03.514625 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.514614 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 21:29:03.514625 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.514623 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 21:29:03.514736 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.514632 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 21:29:03.515387 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.515375 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 21:29:03.515421 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.515391 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 21:29:03.515421 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.515402 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 21:29:03.515421 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.515411 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 21:29:03.515489 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.515429 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 21:29:03.515489 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.515443 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 21:29:03.516797 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.516787 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 21:29:03.516797 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.516798 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 21:29:03.520555 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.520543 2584 server.go:1263] "Started kubelet" Nov 03 21:29:03.520727 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.520638 2584 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 21:29:03.520727 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.520628 2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 21:29:03.520727 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.520711 2584 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 21:29:03.521250 ip-10-0-131-253 systemd[1]: Started Kubernetes Kubelet. Nov 03 21:29:03.523260 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.523238 2584 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 21:29:03.524356 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.524344 2584 server.go:317] "Adding debug handlers to kubelet server" Nov 03 21:29:03.526755 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.526717 2584 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 21:29:03.529440 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.529411 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 21:29:03.529844 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.529825 2584 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 21:29:03.530020 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.529999 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:03.530791 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.530772 2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 21:29:03.530791 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.530782 2584 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 21:29:03.530888 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.530800 2584 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 21:29:03.530888 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.530817 2584 factory.go:153] Registering CRI-O factory Nov 03 21:29:03.530888 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.530831 2584 factory.go:223] Registration of the crio container factory successfully Nov 03 21:29:03.530888 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.530876 2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 21:29:03.530888 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.530883 2584 factory.go:55] Registering systemd factory Nov 03 21:29:03.530888 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.530890 2584 factory.go:223] Registration of the systemd container factory successfully Nov 03 21:29:03.531046 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.530903 2584 factory.go:103] Registering Raw factory Nov 03 21:29:03.531046 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.530911 2584 manager.go:1199] Started watching for new ooms in manager Nov 03 21:29:03.531046 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.530913 2584 reconstruct.go:97] "Volume reconstruction finished" Nov 03 21:29:03.531046 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.530920 2584 reconciler.go:29] "Reconciler: start to sync state" Nov 03 21:29:03.531821 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.531805 2584 manager.go:319] Starting recovery of all containers Nov 03 21:29:03.541570 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.541555 2584 manager.go:324] Recovery completed Nov 03 21:29:03.545811 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.545798 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:29:03.547730 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.547700 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientMemory" Nov 03 21:29:03.547789 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.547743 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 21:29:03.547789 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.547757 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientPID" Nov 03 21:29:03.548193 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.548181 2584 cpu_manager.go:226] "Starting" policy="none" Nov 03 21:29:03.548193 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.548192 2584 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 21:29:03.548290 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.548196 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:29:03.548290 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.548210 2584 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 21:29:03.549867 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.549845 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientMemory" Nov 03 21:29:03.549940 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.549911 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 21:29:03.549940 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.549930 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientPID" Nov 03 21:29:03.550023 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.549974 2584 policy_none.go:50] "Start" Nov 03 21:29:03.550023 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.549993 2584 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 21:29:03.550023 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.550011 2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 21:29:03.551404 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.551390 2584 policy_none.go:44] "Start" Nov 03 21:29:03.558530 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.558503 2584 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 21:29:03.558530 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.558510 2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-253.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 21:29:03.559458 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.558504 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-253.ec2.internal.18e4a63f8f78ec56 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-253.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-253.ec2.internal,},FirstTimestamp:2026-11-03 21:29:03.520517206 +0000 UTC m=+0.412624673,LastTimestamp:2026-11-03 21:29:03.520517206 +0000 UTC m=+0.412624673,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-253.ec2.internal,}" Nov 03 21:29:03.560306 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.560289 2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 21:29:03.572426 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.572398 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lspf7" Nov 03 21:29:03.573790 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.573769 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 21:29:03.573790 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.573768 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 21:29:03.573972 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.573749 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-253.ec2.internal.18e4a63f911825be default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-253.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-253.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-253.ec2.internal,},FirstTimestamp:2026-11-03 21:29:03.547729342 +0000 UTC m=+0.439836811,LastTimestamp:2026-11-03 21:29:03.547729342 +0000 UTC m=+0.439836811,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-253.ec2.internal,}" Nov 03 21:29:03.574031 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.573977 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 21:29:03.599249 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.581867 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lspf7" Nov 03 21:29:03.599249 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.592217 2584 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 21:29:03.599249 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.593295 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-253.ec2.internal.18e4a63f91186e27 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-253.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-253.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-253.ec2.internal,},FirstTimestamp:2026-11-03 21:29:03.547747879 +0000 UTC m=+0.439855346,LastTimestamp:2026-11-03 21:29:03.547747879 +0000 UTC m=+0.439855346,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-253.ec2.internal,}" Nov 03 21:29:03.599249 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.596925 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 21:29:03.599249 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.597983 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 21:29:03.599249 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.598005 2584 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 21:29:03.599249 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.598037 2584 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 21:29:03.599249 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.598074 2584 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 21:29:03.607094 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.607080 2584 manager.go:341] "Starting Device Plugin manager" Nov 03 21:29:03.607188 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.607106 2584 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 21:29:03.607188 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.607115 2584 server.go:87] "Starting device plugin registration server" Nov 03 21:29:03.607382 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.607371 2584 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 21:29:03.607434 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.607383 2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 21:29:03.607497 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.607482 2584 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 21:29:03.607599 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.607575 2584 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 21:29:03.607599 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.607587 2584 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 21:29:03.608036 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.608016 2584 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 21:29:03.608129 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.608063 2584 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:03.613928 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.613912 2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:29:03.698481 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.698420 2584 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal"] Nov 03 21:29:03.698574 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.698489 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:29:03.699438 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.699424 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientMemory" Nov 03 21:29:03.699498 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.699452 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 21:29:03.699621 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.699464 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientPID" Nov 03 21:29:03.701025 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.701013 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:29:03.701205 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.701192 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:03.701267 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.701226 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:29:03.701864 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.701848 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientMemory" Nov 03 21:29:03.701946 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.701878 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 21:29:03.701946 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.701893 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientPID" Nov 03 21:29:03.702041 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.701879 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientMemory" Nov 03 21:29:03.702041 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.701994 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 21:29:03.702041 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.702011 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientPID" Nov 03 21:29:03.703233 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.703216 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" Nov 03 21:29:03.703312 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.703250 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:29:03.703934 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.703920 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientMemory" Nov 03 21:29:03.704012 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.703940 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 21:29:03.704012 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.703952 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientPID" Nov 03 21:29:03.707518 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.707445 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:29:03.708161 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.708146 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientMemory" Nov 03 21:29:03.708227 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.708173 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 21:29:03.708227 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.708194 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientPID" Nov 03 21:29:03.708227 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.708212 2584 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-253.ec2.internal" Nov 03 21:29:03.728999 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.728976 2584 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-253.ec2.internal" Nov 03 21:29:03.728999 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.729002 2584 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-253.ec2.internal\": node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:03.729558 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.729537 2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-253.ec2.internal\" not found" node="ip-10-0-131-253.ec2.internal" Nov 03 21:29:03.733205 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.733188 2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-253.ec2.internal\" not found" node="ip-10-0-131-253.ec2.internal" Nov 03 21:29:03.765978 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.765951 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:03.831693 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.831616 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d42cfc86d4650e909d7de42d9326db55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"d42cfc86d4650e909d7de42d9326db55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:03.831799 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.831716 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d42cfc86d4650e909d7de42d9326db55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"d42cfc86d4650e909d7de42d9326db55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:03.831799 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.831735 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9dff6629a956b5c1c96ca015020c4e26-config\") pod \"kube-apiserver-proxy-ip-10-0-131-253.ec2.internal\" (UID: \"9dff6629a956b5c1c96ca015020c4e26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" Nov 03 21:29:03.866532 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.866510 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:03.932717 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.932691 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d42cfc86d4650e909d7de42d9326db55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"d42cfc86d4650e909d7de42d9326db55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:03.932789 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.932720 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d42cfc86d4650e909d7de42d9326db55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"d42cfc86d4650e909d7de42d9326db55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:03.932789 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.932735 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9dff6629a956b5c1c96ca015020c4e26-config\") pod \"kube-apiserver-proxy-ip-10-0-131-253.ec2.internal\" (UID: \"9dff6629a956b5c1c96ca015020c4e26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" Nov 03 21:29:03.932884 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.932787 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9dff6629a956b5c1c96ca015020c4e26-config\") pod \"kube-apiserver-proxy-ip-10-0-131-253.ec2.internal\" (UID: \"9dff6629a956b5c1c96ca015020c4e26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" Nov 03 21:29:03.932884 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.932792 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d42cfc86d4650e909d7de42d9326db55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"d42cfc86d4650e909d7de42d9326db55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:03.932884 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:03.932813 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d42cfc86d4650e909d7de42d9326db55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"d42cfc86d4650e909d7de42d9326db55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:03.967230 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:03.967174 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:04.033501 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:04.033476 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:04.037496 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:04.037316 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" Nov 03 21:29:04.068180 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:04.068156 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:04.168736 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:04.168694 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:04.269216 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:04.269149 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:04.369721 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:04.369674 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:04.466645 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:04.466612 2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 21:29:04.467148 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:04.466794 2584 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 21:29:04.469838 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:04.469818 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:04.570186 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:04.570116 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:04.573599 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:04.573577 2584 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:29:04.583163 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:04.583134 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42cfc86d4650e909d7de42d9326db55.slice/crio-99472fbf0a315300178bc050744a9e79958553dfdacf7183a13f57657a53a0e6 WatchSource:0}: Error finding container 99472fbf0a315300178bc050744a9e79958553dfdacf7183a13f57657a53a0e6: Status 404 returned error can't find the container with id 99472fbf0a315300178bc050744a9e79958553dfdacf7183a13f57657a53a0e6 Nov 03 21:29:04.588455 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:04.588422 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 21:24:04 +0000 UTC" deadline="2028-04-18 08:36:53.804925059 +0000 UTC" Nov 03 21:29:04.588500 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:04.588460 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12755h7m49.216470116s" Nov 03 21:29:04.593264 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:04.591396 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:29:04.601345 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:04.601303 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" event={"ID":"d42cfc86d4650e909d7de42d9326db55","Type":"ContainerStarted","Data":"99472fbf0a315300178bc050744a9e79958553dfdacf7183a13f57657a53a0e6"} Nov 03 21:29:04.602435 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:04.602415 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" event={"ID":"9dff6629a956b5c1c96ca015020c4e26","Type":"ContainerStarted","Data":"6ba8f748bd56cf066e3fdc0b9b6ed58eec228f3c3edfd37afe21903f60c47644"} Nov 03 21:29:04.670300 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:04.670268 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:04.698538 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:04.698520 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:29:04.699783 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:04.699768 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientMemory" Nov 03 21:29:04.699872 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:04.699795 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 21:29:04.699872 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:04.699808 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientPID" Nov 03 21:29:04.771253 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:04.771223 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:04.872828 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:04.872745 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:04.973842 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:04.973814 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:05.056053 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.056024 2584 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:29:05.130308 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.130236 2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:05.145958 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.145901 2584 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:29:05.155142 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.155114 2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 21:29:05.156405 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.156380 2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" Nov 03 21:29:05.176456 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.176434 2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 21:29:05.504923 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.504839 2584 apiserver.go:52] "Watching apiserver" Nov 03 21:29:05.529897 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.529862 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 21:29:05.534169 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.534144 2584 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 21:29:05.534491 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.534473 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-qvv5p","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv","openshift-multus/multus-gdt8s","openshift-network-diagnostics/network-check-target-pllqd","openshift-ovn-kubernetes/ovnkube-node-ltgjc","kube-system/global-pull-secret-syncer-z5jjd","kube-system/konnectivity-agent-vh4hd","kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal","openshift-cluster-node-tuning-operator/tuned-cvc9c","openshift-dns/node-resolver-5wq87","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal","openshift-image-registry/node-ca-rwd6w","openshift-network-operator/iptables-alerter-rc5dj","openshift-multus/multus-additional-cni-plugins-2lwrg"] Nov 03 21:29:05.537367 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.537343 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:05.539695 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.539658 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.542221 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.542201 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.544574 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.544556 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:05.544667 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:05.544631 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pllqd" podUID="ca003584-cd18-4740-a619-66c89853c773" Nov 03 21:29:05.547356 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.546936 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.549416 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.549396 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.553792 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.552597 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 21:29:05.553792 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.552640 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.553792 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.552677 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vzdjb\"" Nov 03 21:29:05.553792 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.552758 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 21:29:05.554403 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.554140 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 21:29:05.554403 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.554279 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 21:29:05.554403 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.554349 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 21:29:05.560754 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.555014 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w4krc\"" Nov 03 21:29:05.560754 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.555412 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:05.560754 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.555461 2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:29:05.561138 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.561120 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:05.561698 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.561674 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 21:29:05.562052 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.561762 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 21:29:05.562139 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.562095 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 21:29:05.562139 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.562099 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 21:29:05.562139 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.562122 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6tkv6\"" Nov 03 21:29:05.562841 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.562811 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:05.562922 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:05.562885 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvv5p" podUID="2a406a0d-024a-411f-94c2-953a739c214f" Nov 03 21:29:05.565405 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.565384 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:05.568053 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.568036 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:05.568145 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:05.568081 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z5jjd" podUID="58ec685b-55f0-42a2-9112-6fe8a813fd8a" Nov 03 21:29:05.632051 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.632031 2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 21:29:05.642835 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.642806 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysctl-d\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.642950 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.642849 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-k8s-cni-cncf-io\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.642950 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.642929 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-conf-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.643066 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.642992 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d65ba521-c50d-47b4-82d9-1910dde4fbb1-hosts-file\") pod \"node-resolver-5wq87\" (UID: \"d65ba521-c50d-47b4-82d9-1910dde4fbb1\") " pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:05.643066 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643045 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-var-lib-kubelet\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.643176 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643093 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5cfd\" (UniqueName: \"kubernetes.io/projected/c5013527-41c2-4355-9204-594fe21d3c8a-kube-api-access-g5cfd\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.643176 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643144 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqf4v\" (UniqueName: \"kubernetes.io/projected/fb46e9cd-a477-4380-8d82-63a93d0f4099-kube-api-access-mqf4v\") pod \"node-ca-rwd6w\" (UID: \"fb46e9cd-a477-4380-8d82-63a93d0f4099\") " pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:05.643250 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643187 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-cni-bin\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.643250 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643229 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-multus-certs\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.643319 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643261 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9kvm\" (UniqueName: \"kubernetes.io/projected/b04c787c-4de0-483a-9f96-698fa046d04b-kube-api-access-q9kvm\") pod \"iptables-alerter-rc5dj\" (UID: \"b04c787c-4de0-483a-9f96-698fa046d04b\") " pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:05.643319 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643290 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:05.643319 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643314 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-var-lib-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.643439 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643395 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.643439 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643422 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovnkube-script-lib\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.643538 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643438 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-sys\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.643538 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643455 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5013527-41c2-4355-9204-594fe21d3c8a-etc-tuned\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.643538 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643470 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f738a71-6d2b-4ade-ac92-990cf0a2b311-konnectivity-ca\") pod \"konnectivity-agent-vh4hd\" (UID: \"1f738a71-6d2b-4ade-ac92-990cf0a2b311\") " pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:05.643538 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643491 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-etc-kubernetes\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.643538 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643507 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/58ec685b-55f0-42a2-9112-6fe8a813fd8a-kubelet-config\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:05.643538 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643527 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/58ec685b-55f0-42a2-9112-6fe8a813fd8a-dbus\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:05.643804 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643593 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-device-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.643804 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643628 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-systemd-units\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.643804 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643661 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-cni-bin\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.643804 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643695 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.643804 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643738 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b04c787c-4de0-483a-9f96-698fa046d04b-iptables-alerter-script\") pod \"iptables-alerter-rc5dj\" (UID: \"b04c787c-4de0-483a-9f96-698fa046d04b\") " pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:05.643804 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643762 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.643804 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643789 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-log-socket\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.644146 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643820 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-env-overrides\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.644146 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643846 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovn-node-metrics-cert\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.644146 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643875 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-cnibin\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.644146 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643898 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysctl-conf\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.644146 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643921 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-host\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.644146 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643943 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-systemd\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.644146 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643967 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-ovn\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.644146 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.643990 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-system-cni-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.644146 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644016 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb46e9cd-a477-4380-8d82-63a93d0f4099-host\") pod \"node-ca-rwd6w\" (UID: \"fb46e9cd-a477-4380-8d82-63a93d0f4099\") " pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:05.644548 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644186 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.644548 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644209 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/710eeea5-8bbf-4b10-8167-45a61512ef7f-cni-binary-copy\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.644548 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644232 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs\") pod \"network-metrics-daemon-qvv5p\" (UID: \"2a406a0d-024a-411f-94c2-953a739c214f\") " pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:05.644548 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644250 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d65ba521-c50d-47b4-82d9-1910dde4fbb1-tmp-dir\") pod \"node-resolver-5wq87\" (UID: \"d65ba521-c50d-47b4-82d9-1910dde4fbb1\") " pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:05.644548 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644267 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.644548 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644288 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysconfig\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.644548 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644317 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b04c787c-4de0-483a-9f96-698fa046d04b-host-slash\") pod \"iptables-alerter-rc5dj\" (UID: \"b04c787c-4de0-483a-9f96-698fa046d04b\") " pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:05.644548 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644333 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-cni-netd\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.644548 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644348 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.644548 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644369 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-run\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.644548 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644383 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5013527-41c2-4355-9204-594fe21d3c8a-tmp\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.644548 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644405 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb46e9cd-a477-4380-8d82-63a93d0f4099-serviceca\") pod \"node-ca-rwd6w\" (UID: \"fb46e9cd-a477-4380-8d82-63a93d0f4099\") " pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:05.644548 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644420 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-slash\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.644548 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644433 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-run-netns\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.644548 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644453 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovnkube-config\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.644548 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644468 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.645263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644484 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f738a71-6d2b-4ade-ac92-990cf0a2b311-agent-certs\") pod \"konnectivity-agent-vh4hd\" (UID: \"1f738a71-6d2b-4ade-ac92-990cf0a2b311\") " pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:05.645263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644499 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.645263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644557 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf9vr\" (UniqueName: \"kubernetes.io/projected/0d3ddfc6-6d13-4b06-983f-75cca78543ea-kube-api-access-cf9vr\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.645263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644574 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-system-cni-dir\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.645263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644594 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.645263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644610 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-kubernetes\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.645263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644627 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-systemd\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.645263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644641 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-socket-dir-parent\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.645263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644657 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-daemon-config\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.645263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644677 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x526l\" (UniqueName: \"kubernetes.io/projected/1bc36693-102d-48e8-bb01-302b38055e05-kube-api-access-x526l\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.645263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644692 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-modprobe-d\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.645263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644709 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-lib-modules\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.645263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644723 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-cnibin\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.645263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644911 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-netns\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.645263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.644955 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-kubelet\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.645263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.645021 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-hostroot\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.645971 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.645054 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-socket-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.645971 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.645081 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-sys-fs\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.645971 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.645101 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-node-log\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.645971 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.645139 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8bqr\" (UniqueName: \"kubernetes.io/projected/2a406a0d-024a-411f-94c2-953a739c214f-kube-api-access-c8bqr\") pod \"network-metrics-daemon-qvv5p\" (UID: \"2a406a0d-024a-411f-94c2-953a739c214f\") " pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:05.645971 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.645185 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-kubelet\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.645971 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.645206 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-etc-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.645971 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.645230 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-cni-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.645971 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.645311 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-cni-multus\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.645971 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.645356 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwmwp\" (UniqueName: \"kubernetes.io/projected/710eeea5-8bbf-4b10-8167-45a61512ef7f-kube-api-access-pwmwp\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.645971 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.645394 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7dl8\" (UniqueName: \"kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8\") pod \"network-check-target-pllqd\" (UID: \"ca003584-cd18-4740-a619-66c89853c773\") " pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:05.645971 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.645419 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhwr2\" (UniqueName: \"kubernetes.io/projected/d65ba521-c50d-47b4-82d9-1910dde4fbb1-kube-api-access-rhwr2\") pod \"node-resolver-5wq87\" (UID: \"d65ba521-c50d-47b4-82d9-1910dde4fbb1\") " pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:05.645971 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.645441 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-registration-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.645971 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.645469 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6jjd\" (UniqueName: \"kubernetes.io/projected/add088ee-8843-4be8-ad89-f396481163c5-kube-api-access-w6jjd\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.662214 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.662181 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 21:29:05.662214 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.662179 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:29:05.662474 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.662196 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 21:29:05.662536 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.662485 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 21:29:05.662536 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.662486 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 21:29:05.662645 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.662536 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 21:29:05.662645 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.662611 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 21:29:05.662769 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.662657 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xgwf4\"" Nov 03 21:29:05.662769 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.662699 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4tzc5\"" Nov 03 21:29:05.662869 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.662838 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 21:29:05.662918 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.662874 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 21:29:05.662918 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.662893 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 21:29:05.662918 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.662894 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 21:29:05.663064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.663032 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 21:29:05.663064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.663037 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tnnmd\"" Nov 03 21:29:05.663157 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.663041 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 21:29:05.663157 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.663135 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 21:29:05.663252 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.663189 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 21:29:05.663300 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.663256 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d5mwz\"" Nov 03 21:29:05.663373 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.663356 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:29:05.663427 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.663377 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mhg7x\"" Nov 03 21:29:05.663478 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.663429 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 21:29:05.663577 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.663566 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6tvtx\"" Nov 03 21:29:05.663774 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.663758 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 21:29:05.671630 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.671588 2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 21:29:05.671842 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.671701 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-twvvs" Nov 03 21:29:05.746565 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746524 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/58ec685b-55f0-42a2-9112-6fe8a813fd8a-kubelet-config\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:05.746777 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746570 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/58ec685b-55f0-42a2-9112-6fe8a813fd8a-dbus\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:05.746777 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746604 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-device-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.746777 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746630 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-systemd-units\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.746777 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746656 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-cni-bin\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.746777 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746682 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.746777 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746726 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b04c787c-4de0-483a-9f96-698fa046d04b-iptables-alerter-script\") pod \"iptables-alerter-rc5dj\" (UID: \"b04c787c-4de0-483a-9f96-698fa046d04b\") " pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:05.746777 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746757 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.747119 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746785 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-log-socket\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.747119 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746810 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-env-overrides\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.747119 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746837 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovn-node-metrics-cert\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.747119 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746861 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-cnibin\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.747119 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746884 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysctl-conf\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.747119 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746911 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-host\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.747119 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746935 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-systemd\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.747119 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746957 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-ovn\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.747119 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.746982 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-system-cni-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.747119 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747007 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb46e9cd-a477-4380-8d82-63a93d0f4099-host\") pod \"node-ca-rwd6w\" (UID: \"fb46e9cd-a477-4380-8d82-63a93d0f4099\") " pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:05.747507 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747479 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.747507 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747509 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/710eeea5-8bbf-4b10-8167-45a61512ef7f-cni-binary-copy\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.747686 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747527 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs\") pod \"network-metrics-daemon-qvv5p\" (UID: \"2a406a0d-024a-411f-94c2-953a739c214f\") " pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:05.747686 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747543 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d65ba521-c50d-47b4-82d9-1910dde4fbb1-tmp-dir\") pod \"node-resolver-5wq87\" (UID: \"d65ba521-c50d-47b4-82d9-1910dde4fbb1\") " pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:05.747686 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747558 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.747686 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747574 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysconfig\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.747686 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747597 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b04c787c-4de0-483a-9f96-698fa046d04b-host-slash\") pod \"iptables-alerter-rc5dj\" (UID: \"b04c787c-4de0-483a-9f96-698fa046d04b\") " pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:05.747686 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747613 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-cni-netd\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.747686 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747628 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.747686 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747642 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-run\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.747686 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747656 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5013527-41c2-4355-9204-594fe21d3c8a-tmp\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.747686 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747676 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb46e9cd-a477-4380-8d82-63a93d0f4099-serviceca\") pod \"node-ca-rwd6w\" (UID: \"fb46e9cd-a477-4380-8d82-63a93d0f4099\") " pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:05.748013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747694 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-slash\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.748013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747708 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-run-netns\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.748013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747722 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovnkube-config\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.748013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747753 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.748013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747777 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f738a71-6d2b-4ade-ac92-990cf0a2b311-agent-certs\") pod \"konnectivity-agent-vh4hd\" (UID: \"1f738a71-6d2b-4ade-ac92-990cf0a2b311\") " pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:05.748013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747794 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.748013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747809 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf9vr\" (UniqueName: \"kubernetes.io/projected/0d3ddfc6-6d13-4b06-983f-75cca78543ea-kube-api-access-cf9vr\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.748013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747824 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-system-cni-dir\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.748013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747841 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.748013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747890 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-kubernetes\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.748013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747905 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-systemd\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.748013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747925 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-socket-dir-parent\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.748013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747949 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-daemon-config\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.748013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747965 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x526l\" (UniqueName: \"kubernetes.io/projected/1bc36693-102d-48e8-bb01-302b38055e05-kube-api-access-x526l\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.748013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.747981 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-modprobe-d\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.748013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748004 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-lib-modules\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748023 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-cnibin\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748043 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-netns\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748058 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-kubelet\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748081 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-hostroot\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748098 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-socket-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748120 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-sys-fs\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748135 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-node-log\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748152 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bqr\" (UniqueName: \"kubernetes.io/projected/2a406a0d-024a-411f-94c2-953a739c214f-kube-api-access-c8bqr\") pod \"network-metrics-daemon-qvv5p\" (UID: \"2a406a0d-024a-411f-94c2-953a739c214f\") " pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748167 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-kubelet\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748183 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-etc-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748206 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-cni-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748221 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-cni-multus\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748235 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmwp\" (UniqueName: \"kubernetes.io/projected/710eeea5-8bbf-4b10-8167-45a61512ef7f-kube-api-access-pwmwp\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748252 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dl8\" (UniqueName: \"kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8\") pod \"network-check-target-pllqd\" (UID: \"ca003584-cd18-4740-a619-66c89853c773\") " pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748267 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhwr2\" (UniqueName: \"kubernetes.io/projected/d65ba521-c50d-47b4-82d9-1910dde4fbb1-kube-api-access-rhwr2\") pod \"node-resolver-5wq87\" (UID: \"d65ba521-c50d-47b4-82d9-1910dde4fbb1\") " pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748287 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-registration-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748305 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6jjd\" (UniqueName: \"kubernetes.io/projected/add088ee-8843-4be8-ad89-f396481163c5-kube-api-access-w6jjd\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748320 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysctl-d\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748335 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-k8s-cni-cncf-io\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748354 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-conf-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748376 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d65ba521-c50d-47b4-82d9-1910dde4fbb1-hosts-file\") pod \"node-resolver-5wq87\" (UID: \"d65ba521-c50d-47b4-82d9-1910dde4fbb1\") " pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748392 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-var-lib-kubelet\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748407 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5cfd\" (UniqueName: \"kubernetes.io/projected/c5013527-41c2-4355-9204-594fe21d3c8a-kube-api-access-g5cfd\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748423 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqf4v\" (UniqueName: \"kubernetes.io/projected/fb46e9cd-a477-4380-8d82-63a93d0f4099-kube-api-access-mqf4v\") pod \"node-ca-rwd6w\" (UID: \"fb46e9cd-a477-4380-8d82-63a93d0f4099\") " pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748437 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-cni-bin\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748456 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-multus-certs\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748475 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9kvm\" (UniqueName: \"kubernetes.io/projected/b04c787c-4de0-483a-9f96-698fa046d04b-kube-api-access-q9kvm\") pod \"iptables-alerter-rc5dj\" (UID: \"b04c787c-4de0-483a-9f96-698fa046d04b\") " pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748491 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748506 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-var-lib-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748521 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748537 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovnkube-script-lib\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748556 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-sys\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748571 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5013527-41c2-4355-9204-594fe21d3c8a-etc-tuned\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.749064 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748584 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f738a71-6d2b-4ade-ac92-990cf0a2b311-konnectivity-ca\") pod \"konnectivity-agent-vh4hd\" (UID: \"1f738a71-6d2b-4ade-ac92-990cf0a2b311\") " pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748599 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-etc-kubernetes\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748685 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-etc-kubernetes\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748752 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/58ec685b-55f0-42a2-9112-6fe8a813fd8a-kubelet-config\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748821 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-systemd\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748912 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-socket-dir-parent\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:05.748915 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.748985 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-cnibin\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:05.749019 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs podName:2a406a0d-024a-411f-94c2-953a739c214f nodeName:}" failed. No retries permitted until 2026-11-03 21:29:06.248976721 +0000 UTC m=+3.141084197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs") pod "network-metrics-daemon-qvv5p" (UID: "2a406a0d-024a-411f-94c2-953a739c214f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.749161 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysctl-conf\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.749178 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d65ba521-c50d-47b4-82d9-1910dde4fbb1-tmp-dir\") pod \"node-resolver-5wq87\" (UID: \"d65ba521-c50d-47b4-82d9-1910dde4fbb1\") " pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.749211 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-host\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.749247 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysconfig\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.749252 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-ovn\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.749289 2584 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.749302 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-system-cni-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.749354 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb46e9cd-a477-4380-8d82-63a93d0f4099-host\") pod \"node-ca-rwd6w\" (UID: \"fb46e9cd-a477-4380-8d82-63a93d0f4099\") " pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.749390 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b04c787c-4de0-483a-9f96-698fa046d04b-host-slash\") pod \"iptables-alerter-rc5dj\" (UID: \"b04c787c-4de0-483a-9f96-698fa046d04b\") " pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:05.749660 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.749399 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.750397 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.749440 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-cni-netd\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.750397 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.749825 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-daemon-config\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.750397 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.749843 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovnkube-config\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.750397 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.749915 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-systemd\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.750397 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.749943 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/710eeea5-8bbf-4b10-8167-45a61512ef7f-cni-binary-copy\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.750397 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.750137 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.750397 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.750222 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-modprobe-d\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.750397 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.750325 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-lib-modules\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.750397 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.750372 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.750397 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.750382 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-cnibin\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.750840 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.750437 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-netns\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.750840 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.750485 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-kubelet\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.750840 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.750523 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-hostroot\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.750840 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.750581 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb46e9cd-a477-4380-8d82-63a93d0f4099-serviceca\") pod \"node-ca-rwd6w\" (UID: \"fb46e9cd-a477-4380-8d82-63a93d0f4099\") " pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:05.750840 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.750617 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-socket-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.750840 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.750644 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-slash\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.750840 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.750675 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-sys-fs\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.750840 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.750692 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-system-cni-dir\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.750840 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.750716 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-node-log\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.750840 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.750780 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.751306 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.750968 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-kubelet\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.751306 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.751022 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-etc-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.751306 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.751072 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-cni-bin\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.751306 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.751110 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-multus-certs\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.751306 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.751189 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/58ec685b-55f0-42a2-9112-6fe8a813fd8a-dbus\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:05.751306 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.751270 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-systemd-units\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.751610 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.751315 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-cni-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.751610 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.751328 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-cni-bin\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.751610 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.751374 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-device-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.751610 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.751376 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-var-lib-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.751610 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.751422 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.751610 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.751441 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-cni-multus\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.752615 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.751779 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-run\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.752615 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.749295 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-run-netns\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.752615 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.751859 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-sys\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.752615 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.751913 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-env-overrides\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.752615 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.751929 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.752615 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.751928 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d65ba521-c50d-47b4-82d9-1910dde4fbb1-hosts-file\") pod \"node-resolver-5wq87\" (UID: \"d65ba521-c50d-47b4-82d9-1910dde4fbb1\") " pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:05.752615 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.752003 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-registration-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.752615 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.752169 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-k8s-cni-cncf-io\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.752615 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.752171 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysctl-d\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.752615 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.752242 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-var-lib-kubelet\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.752615 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.752264 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-log-socket\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.752615 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.752326 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.752615 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.752367 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-conf-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.752615 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.752427 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-kubernetes\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.753381 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.753038 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f738a71-6d2b-4ade-ac92-990cf0a2b311-konnectivity-ca\") pod \"konnectivity-agent-vh4hd\" (UID: \"1f738a71-6d2b-4ade-ac92-990cf0a2b311\") " pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:05.753381 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:05.753264 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:29:05.753381 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:05.753317 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret podName:58ec685b-55f0-42a2-9112-6fe8a813fd8a nodeName:}" failed. No retries permitted until 2026-11-03 21:29:06.253300253 +0000 UTC m=+3.145407725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret") pod "global-pull-secret-syncer-z5jjd" (UID: "58ec685b-55f0-42a2-9112-6fe8a813fd8a") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:29:05.762874 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.753840 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.762874 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.753867 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b04c787c-4de0-483a-9f96-698fa046d04b-iptables-alerter-script\") pod \"iptables-alerter-rc5dj\" (UID: \"b04c787c-4de0-483a-9f96-698fa046d04b\") " pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:05.762874 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.754198 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovnkube-script-lib\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.762874 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.754213 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovn-node-metrics-cert\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.762874 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.754357 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.762874 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.754530 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f738a71-6d2b-4ade-ac92-990cf0a2b311-agent-certs\") pod \"konnectivity-agent-vh4hd\" (UID: \"1f738a71-6d2b-4ade-ac92-990cf0a2b311\") " pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:05.762874 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.755124 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5013527-41c2-4355-9204-594fe21d3c8a-tmp\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.762874 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.756761 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5013527-41c2-4355-9204-594fe21d3c8a-etc-tuned\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.779797 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.772821 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x526l\" (UniqueName: \"kubernetes.io/projected/1bc36693-102d-48e8-bb01-302b38055e05-kube-api-access-x526l\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.779797 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.772829 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6jjd\" (UniqueName: \"kubernetes.io/projected/add088ee-8843-4be8-ad89-f396481163c5-kube-api-access-w6jjd\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.779797 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:05.773101 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:29:05.779797 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:05.773121 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:29:05.779797 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:05.773136 2584 projected.go:196] Error preparing data for projected volume kube-api-access-d7dl8 for pod openshift-network-diagnostics/network-check-target-pllqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:29:05.779797 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:05.773230 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8 podName:ca003584-cd18-4740-a619-66c89853c773 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:06.273211825 +0000 UTC m=+3.165319284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d7dl8" (UniqueName: "kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8") pod "network-check-target-pllqd" (UID: "ca003584-cd18-4740-a619-66c89853c773") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:29:05.779797 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.776363 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqf4v\" (UniqueName: \"kubernetes.io/projected/fb46e9cd-a477-4380-8d82-63a93d0f4099-kube-api-access-mqf4v\") pod \"node-ca-rwd6w\" (UID: \"fb46e9cd-a477-4380-8d82-63a93d0f4099\") " pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:05.779797 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.776423 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9kvm\" (UniqueName: \"kubernetes.io/projected/b04c787c-4de0-483a-9f96-698fa046d04b-kube-api-access-q9kvm\") pod \"iptables-alerter-rc5dj\" (UID: \"b04c787c-4de0-483a-9f96-698fa046d04b\") " pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:05.779797 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.776611 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwmwp\" (UniqueName: \"kubernetes.io/projected/710eeea5-8bbf-4b10-8167-45a61512ef7f-kube-api-access-pwmwp\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.779797 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.776689 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhwr2\" (UniqueName: \"kubernetes.io/projected/d65ba521-c50d-47b4-82d9-1910dde4fbb1-kube-api-access-rhwr2\") pod \"node-resolver-5wq87\" (UID: \"d65ba521-c50d-47b4-82d9-1910dde4fbb1\") " pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:05.779797 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.778063 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bqr\" (UniqueName: \"kubernetes.io/projected/2a406a0d-024a-411f-94c2-953a739c214f-kube-api-access-c8bqr\") pod \"network-metrics-daemon-qvv5p\" (UID: \"2a406a0d-024a-411f-94c2-953a739c214f\") " pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:05.779797 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.778313 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf9vr\" (UniqueName: \"kubernetes.io/projected/0d3ddfc6-6d13-4b06-983f-75cca78543ea-kube-api-access-cf9vr\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.779797 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.778511 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5cfd\" (UniqueName: \"kubernetes.io/projected/c5013527-41c2-4355-9204-594fe21d3c8a-kube-api-access-g5cfd\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.853926 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.853889 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:05.864270 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.864106 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:05.866987 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:05.866933 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65ba521_c50d_47b4_82d9_1910dde4fbb1.slice/crio-d8e1d8d876cf1f51ccd3d3f29393e4bd36910f47c12d4a33fdeda7745f288abe WatchSource:0}: Error finding container d8e1d8d876cf1f51ccd3d3f29393e4bd36910f47c12d4a33fdeda7745f288abe: Status 404 returned error can't find the container with id d8e1d8d876cf1f51ccd3d3f29393e4bd36910f47c12d4a33fdeda7745f288abe Nov 03 21:29:05.871023 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.871000 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gdt8s" Nov 03 21:29:05.874121 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:05.874100 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc36693_102d_48e8_bb01_302b38055e05.slice/crio-34f787a783a293e42de8ee6732122662b4dc128e93addec24f269ec8d904f624 WatchSource:0}: Error finding container 34f787a783a293e42de8ee6732122662b4dc128e93addec24f269ec8d904f624: Status 404 returned error can't find the container with id 34f787a783a293e42de8ee6732122662b4dc128e93addec24f269ec8d904f624 Nov 03 21:29:05.882009 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:05.881976 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710eeea5_8bbf_4b10_8167_45a61512ef7f.slice/crio-6be4ba4998986958b5c3e9d233284bf5f5541d8ed2bdd3f82aea70baeba5e738 WatchSource:0}: Error finding container 6be4ba4998986958b5c3e9d233284bf5f5541d8ed2bdd3f82aea70baeba5e738: Status 404 returned error can't find the container with id 6be4ba4998986958b5c3e9d233284bf5f5541d8ed2bdd3f82aea70baeba5e738 Nov 03 21:29:05.902439 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.902413 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:05.906468 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.906337 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:05.908934 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.908914 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:05.919280 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.919101 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:05.920565 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.920265 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:05.920565 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:05.920453 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3ddfc6_6d13_4b06_983f_75cca78543ea.slice/crio-23b21928d442df069c187793509633da76f70ed3ff5f3f305c40e5b60e8e45f2 WatchSource:0}: Error finding container 23b21928d442df069c187793509633da76f70ed3ff5f3f305c40e5b60e8e45f2: Status 404 returned error can't find the container with id 23b21928d442df069c187793509633da76f70ed3ff5f3f305c40e5b60e8e45f2 Nov 03 21:29:05.921968 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:05.921939 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb46e9cd_a477_4380_8d82_63a93d0f4099.slice/crio-b6d02155668d60cb4a8d28bc7248b95465777dc8d30d1454526682d1c47663b6 WatchSource:0}: Error finding container b6d02155668d60cb4a8d28bc7248b95465777dc8d30d1454526682d1c47663b6: Status 404 returned error can't find the container with id b6d02155668d60cb4a8d28bc7248b95465777dc8d30d1454526682d1c47663b6 Nov 03 21:29:05.923766 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:05.923687 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:05.932034 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:05.931997 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd088ee_8843_4be8_ad89_f396481163c5.slice/crio-8d22691ae89709aea408b145b52fd3ea07b74372958366203de5ae2e48bf3c41 WatchSource:0}: Error finding container 8d22691ae89709aea408b145b52fd3ea07b74372958366203de5ae2e48bf3c41: Status 404 returned error can't find the container with id 8d22691ae89709aea408b145b52fd3ea07b74372958366203de5ae2e48bf3c41 Nov 03 21:29:05.936960 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:05.936933 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f738a71_6d2b_4ade_ac92_990cf0a2b311.slice/crio-a5273bfd1b9cfa16e00110e1fae1986c072514af1ac6e63826edc3d0f4fc95a4 WatchSource:0}: Error finding container a5273bfd1b9cfa16e00110e1fae1986c072514af1ac6e63826edc3d0f4fc95a4: Status 404 returned error can't find the container with id a5273bfd1b9cfa16e00110e1fae1986c072514af1ac6e63826edc3d0f4fc95a4 Nov 03 21:29:05.941928 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:05.941879 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04c787c_4de0_483a_9f96_698fa046d04b.slice/crio-73b55eafe425c791994c391877e8fc3b4d2d910a6cb9455e82c5ab9db604ddea WatchSource:0}: Error finding container 73b55eafe425c791994c391877e8fc3b4d2d910a6cb9455e82c5ab9db604ddea: Status 404 returned error can't find the container with id 73b55eafe425c791994c391877e8fc3b4d2d910a6cb9455e82c5ab9db604ddea Nov 03 21:29:06.253050 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.253021 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs\") pod \"network-metrics-daemon-qvv5p\" (UID: \"2a406a0d-024a-411f-94c2-953a739c214f\") " pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:06.253155 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:06.253142 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:29:06.253213 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:06.253194 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs podName:2a406a0d-024a-411f-94c2-953a739c214f nodeName:}" failed. No retries permitted until 2026-11-03 21:29:07.253181643 +0000 UTC m=+4.145289100 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs") pod "network-metrics-daemon-qvv5p" (UID: "2a406a0d-024a-411f-94c2-953a739c214f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:29:06.353671 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.353647 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dl8\" (UniqueName: \"kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8\") pod \"network-check-target-pllqd\" (UID: \"ca003584-cd18-4740-a619-66c89853c773\") " pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:06.353795 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.353706 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:06.353857 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:06.353833 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:29:06.353905 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:06.353885 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret podName:58ec685b-55f0-42a2-9112-6fe8a813fd8a nodeName:}" failed. No retries permitted until 2026-11-03 21:29:07.353866732 +0000 UTC m=+4.245974191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret") pod "global-pull-secret-syncer-z5jjd" (UID: "58ec685b-55f0-42a2-9112-6fe8a813fd8a") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:29:06.354240 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:06.354225 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:29:06.354306 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:06.354244 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:29:06.354306 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:06.354255 2584 projected.go:196] Error preparing data for projected volume kube-api-access-d7dl8 for pod openshift-network-diagnostics/network-check-target-pllqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:29:06.354306 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:06.354301 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8 podName:ca003584-cd18-4740-a619-66c89853c773 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:07.354286821 +0000 UTC m=+4.246394283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7dl8" (UniqueName: "kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8") pod "network-check-target-pllqd" (UID: "ca003584-cd18-4740-a619-66c89853c773") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:29:06.599314 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.598510 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:06.599314 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:06.598910 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvv5p" podUID="2a406a0d-024a-411f-94c2-953a739c214f" Nov 03 21:29:06.620685 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.620467 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" event={"ID":"0d3ddfc6-6d13-4b06-983f-75cca78543ea","Type":"ContainerStarted","Data":"23b21928d442df069c187793509633da76f70ed3ff5f3f305c40e5b60e8e45f2"} Nov 03 21:29:06.624524 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.624447 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" event={"ID":"c5013527-41c2-4355-9204-594fe21d3c8a","Type":"ContainerStarted","Data":"e5a05a9a83ca3195c71150f8bb529a52dbb886a3af847ff62318445e8e1dda4a"} Nov 03 21:29:06.629446 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.628758 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" event={"ID":"9dff6629a956b5c1c96ca015020c4e26","Type":"ContainerStarted","Data":"1dfea53a3b721e82670647ce2da5f216091e49eb2a9c498d6a63195349fb1e99"} Nov 03 21:29:06.646100 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.645999 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gdt8s" event={"ID":"710eeea5-8bbf-4b10-8167-45a61512ef7f","Type":"ContainerStarted","Data":"6be4ba4998986958b5c3e9d233284bf5f5541d8ed2bdd3f82aea70baeba5e738"} Nov 03 21:29:06.648497 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.648418 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" event={"ID":"1bc36693-102d-48e8-bb01-302b38055e05","Type":"ContainerStarted","Data":"34f787a783a293e42de8ee6732122662b4dc128e93addec24f269ec8d904f624"} Nov 03 21:29:06.649640 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.649554 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5wq87" event={"ID":"d65ba521-c50d-47b4-82d9-1910dde4fbb1","Type":"ContainerStarted","Data":"d8e1d8d876cf1f51ccd3d3f29393e4bd36910f47c12d4a33fdeda7745f288abe"} Nov 03 21:29:06.655572 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.654746 2584 generic.go:356] "Generic (PLEG): container finished" podID="d42cfc86d4650e909d7de42d9326db55" containerID="68d436197fa23503099d082adec91ff654e3de302580cae3bd2eb540204197e1" exitCode=0 Nov 03 21:29:06.655572 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.654815 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" event={"ID":"d42cfc86d4650e909d7de42d9326db55","Type":"ContainerDied","Data":"68d436197fa23503099d082adec91ff654e3de302580cae3bd2eb540204197e1"} Nov 03 21:29:06.657954 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.657888 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rc5dj" event={"ID":"b04c787c-4de0-483a-9f96-698fa046d04b","Type":"ContainerStarted","Data":"73b55eafe425c791994c391877e8fc3b4d2d910a6cb9455e82c5ab9db604ddea"} Nov 03 21:29:06.666561 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.666537 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vh4hd" event={"ID":"1f738a71-6d2b-4ade-ac92-990cf0a2b311","Type":"ContainerStarted","Data":"a5273bfd1b9cfa16e00110e1fae1986c072514af1ac6e63826edc3d0f4fc95a4"} Nov 03 21:29:06.668189 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.667032 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" podStartSLOduration=0.667017075 podStartE2EDuration="667.017075ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:06.66680109 +0000 UTC m=+3.558908570" watchObservedRunningTime="2026-11-03 21:29:06.667017075 +0000 UTC m=+3.559124555" Nov 03 21:29:06.669282 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.669145 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lwrg" event={"ID":"add088ee-8843-4be8-ad89-f396481163c5","Type":"ContainerStarted","Data":"8d22691ae89709aea408b145b52fd3ea07b74372958366203de5ae2e48bf3c41"} Nov 03 21:29:06.672085 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.672063 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rwd6w" event={"ID":"fb46e9cd-a477-4380-8d82-63a93d0f4099","Type":"ContainerStarted","Data":"b6d02155668d60cb4a8d28bc7248b95465777dc8d30d1454526682d1c47663b6"} Nov 03 21:29:06.676086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.675974 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 21:24:06 +0000 UTC" deadline="2028-06-17 04:12:24.431953272 +0000 UTC" Nov 03 21:29:06.676086 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:06.676027 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14190h43m17.755930225s" Nov 03 21:29:07.263309 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:07.263208 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs\") pod \"network-metrics-daemon-qvv5p\" (UID: \"2a406a0d-024a-411f-94c2-953a739c214f\") " pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:07.263488 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:07.263414 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:29:07.263488 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:07.263483 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs podName:2a406a0d-024a-411f-94c2-953a739c214f nodeName:}" failed. No retries permitted until 2026-11-03 21:29:09.263461106 +0000 UTC m=+6.155568579 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs") pod "network-metrics-daemon-qvv5p" (UID: "2a406a0d-024a-411f-94c2-953a739c214f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:29:07.363761 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:07.363701 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dl8\" (UniqueName: \"kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8\") pod \"network-check-target-pllqd\" (UID: \"ca003584-cd18-4740-a619-66c89853c773\") " pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:07.363761 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:07.363766 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:07.364014 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:07.363920 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:29:07.364014 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:07.363989 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret podName:58ec685b-55f0-42a2-9112-6fe8a813fd8a nodeName:}" failed. No retries permitted until 2026-11-03 21:29:09.363967921 +0000 UTC m=+6.256075393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret") pod "global-pull-secret-syncer-z5jjd" (UID: "58ec685b-55f0-42a2-9112-6fe8a813fd8a") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:29:07.364462 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:07.364438 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:29:07.364462 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:07.364466 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:29:07.364633 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:07.364480 2584 projected.go:196] Error preparing data for projected volume kube-api-access-d7dl8 for pod openshift-network-diagnostics/network-check-target-pllqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:29:07.364633 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:07.364526 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8 podName:ca003584-cd18-4740-a619-66c89853c773 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:09.364511179 +0000 UTC m=+6.256618640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7dl8" (UniqueName: "kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8") pod "network-check-target-pllqd" (UID: "ca003584-cd18-4740-a619-66c89853c773") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:29:07.602436 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:07.602351 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:07.602931 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:07.602489 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pllqd" podUID="ca003584-cd18-4740-a619-66c89853c773" Nov 03 21:29:07.602931 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:07.602903 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:07.603060 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:07.603006 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z5jjd" podUID="58ec685b-55f0-42a2-9112-6fe8a813fd8a" Nov 03 21:29:07.676897 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:07.676826 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 21:24:06 +0000 UTC" deadline="2028-03-30 06:11:58.192036751 +0000 UTC" Nov 03 21:29:07.676897 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:07.676866 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12296h42m50.515175837s" Nov 03 21:29:07.697520 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:07.697486 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" event={"ID":"d42cfc86d4650e909d7de42d9326db55","Type":"ContainerStarted","Data":"372f8cf78df34651a8fa7f2fe91692c75cebd18e33f4ab9ac3d4ee8ddc408ae6"} Nov 03 21:29:08.600058 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:08.599269 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:08.600058 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:08.599444 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvv5p" podUID="2a406a0d-024a-411f-94c2-953a739c214f" Nov 03 21:29:09.292084 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:09.292042 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs\") pod \"network-metrics-daemon-qvv5p\" (UID: \"2a406a0d-024a-411f-94c2-953a739c214f\") " pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:09.292586 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:09.292231 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:29:09.292586 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:09.292297 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs podName:2a406a0d-024a-411f-94c2-953a739c214f nodeName:}" failed. No retries permitted until 2026-11-03 21:29:13.292278448 +0000 UTC m=+10.184385921 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs") pod "network-metrics-daemon-qvv5p" (UID: "2a406a0d-024a-411f-94c2-953a739c214f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:29:09.393287 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:09.393244 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dl8\" (UniqueName: \"kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8\") pod \"network-check-target-pllqd\" (UID: \"ca003584-cd18-4740-a619-66c89853c773\") " pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:09.393472 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:09.393308 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:09.393472 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:09.393447 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:29:09.393586 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:09.393518 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret podName:58ec685b-55f0-42a2-9112-6fe8a813fd8a nodeName:}" failed. No retries permitted until 2026-11-03 21:29:13.393498427 +0000 UTC m=+10.285605885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret") pod "global-pull-secret-syncer-z5jjd" (UID: "58ec685b-55f0-42a2-9112-6fe8a813fd8a") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:29:09.393986 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:09.393961 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:29:09.393986 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:09.393986 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:29:09.394137 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:09.394003 2584 projected.go:196] Error preparing data for projected volume kube-api-access-d7dl8 for pod openshift-network-diagnostics/network-check-target-pllqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:29:09.394137 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:09.394054 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8 podName:ca003584-cd18-4740-a619-66c89853c773 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:13.394037722 +0000 UTC m=+10.286145203 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7dl8" (UniqueName: "kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8") pod "network-check-target-pllqd" (UID: "ca003584-cd18-4740-a619-66c89853c773") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:29:09.601943 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:09.601869 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:09.602090 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:09.602008 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pllqd" podUID="ca003584-cd18-4740-a619-66c89853c773" Nov 03 21:29:09.602090 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:09.602082 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:09.602192 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:09.602172 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z5jjd" podUID="58ec685b-55f0-42a2-9112-6fe8a813fd8a" Nov 03 21:29:09.720917 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:09.720877 2584 generic.go:356] "Generic (PLEG): container finished" podID="add088ee-8843-4be8-ad89-f396481163c5" containerID="35910c1394eb957219207d7a4230dca0cf6bc97ce0474eaaa92473e10edf36f4" exitCode=0 Nov 03 21:29:09.721098 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:09.720940 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lwrg" event={"ID":"add088ee-8843-4be8-ad89-f396481163c5","Type":"ContainerDied","Data":"35910c1394eb957219207d7a4230dca0cf6bc97ce0474eaaa92473e10edf36f4"} Nov 03 21:29:09.792813 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:09.792439 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" podStartSLOduration=3.792421091 podStartE2EDuration="3.792421091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:07.736973891 +0000 UTC m=+4.629081372" watchObservedRunningTime="2026-11-03 21:29:09.792421091 +0000 UTC m=+6.684528574" Nov 03 21:29:10.598798 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:10.598666 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:10.599298 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:10.598826 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvv5p" podUID="2a406a0d-024a-411f-94c2-953a739c214f" Nov 03 21:29:10.735882 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:10.735842 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vh4hd" event={"ID":"1f738a71-6d2b-4ade-ac92-990cf0a2b311","Type":"ContainerStarted","Data":"a64fa846dc7e68cf5f89ad5d4fefe311654687a8b37cb84c718bb115840ad76e"} Nov 03 21:29:11.602261 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:11.601157 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:11.602261 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:11.601282 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pllqd" podUID="ca003584-cd18-4740-a619-66c89853c773" Nov 03 21:29:11.602261 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:11.601694 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:11.602261 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:11.601825 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z5jjd" podUID="58ec685b-55f0-42a2-9112-6fe8a813fd8a" Nov 03 21:29:11.741756 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:11.741451 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rwd6w" event={"ID":"fb46e9cd-a477-4380-8d82-63a93d0f4099","Type":"ContainerStarted","Data":"eb0fead63682aabeeaa620a63fc6c75afbc615ec432294d4a5c02b286e48b0b4"} Nov 03 21:29:11.748465 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:11.748433 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" event={"ID":"1bc36693-102d-48e8-bb01-302b38055e05","Type":"ContainerStarted","Data":"596a890a579f4f70be6ba5b9e4bc6040695b43af4753383895b81301601fb6c2"} Nov 03 21:29:11.795721 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:11.795614 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vh4hd" podStartSLOduration=2.39243754 podStartE2EDuration="6.795593546s" totalImagesPullingTime="4.403156006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:10.7766494 +0000 UTC m=+7.668756983" watchObservedRunningTime="2026-11-03 21:29:11.795593546 +0000 UTC m=+8.687701027" Nov 03 21:29:12.598890 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:12.598816 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:12.599058 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:12.598960 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvv5p" podUID="2a406a0d-024a-411f-94c2-953a739c214f" Nov 03 21:29:13.213934 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:13.213888 2584 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 21:29:13.328121 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:13.328035 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs\") pod \"network-metrics-daemon-qvv5p\" (UID: \"2a406a0d-024a-411f-94c2-953a739c214f\") " pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:13.328330 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:13.328309 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:29:13.328396 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:13.328388 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs podName:2a406a0d-024a-411f-94c2-953a739c214f nodeName:}" failed. No retries permitted until 2026-11-03 21:29:21.328369491 +0000 UTC m=+18.220476972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs") pod "network-metrics-daemon-qvv5p" (UID: "2a406a0d-024a-411f-94c2-953a739c214f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:29:13.429353 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:13.428513 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dl8\" (UniqueName: \"kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8\") pod \"network-check-target-pllqd\" (UID: \"ca003584-cd18-4740-a619-66c89853c773\") " pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:13.429353 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:13.428577 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:13.429353 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:13.428765 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:29:13.429353 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:13.428835 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret podName:58ec685b-55f0-42a2-9112-6fe8a813fd8a nodeName:}" failed. No retries permitted until 2026-11-03 21:29:21.4288161 +0000 UTC m=+18.320923576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret") pod "global-pull-secret-syncer-z5jjd" (UID: "58ec685b-55f0-42a2-9112-6fe8a813fd8a") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:29:13.429353 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:13.428929 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:29:13.429353 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:13.428944 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:29:13.429353 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:13.428958 2584 projected.go:196] Error preparing data for projected volume kube-api-access-d7dl8 for pod openshift-network-diagnostics/network-check-target-pllqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:29:13.429353 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:13.428995 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8 podName:ca003584-cd18-4740-a619-66c89853c773 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:21.428981684 +0000 UTC m=+18.321089155 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7dl8" (UniqueName: "kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8") pod "network-check-target-pllqd" (UID: "ca003584-cd18-4740-a619-66c89853c773") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:29:13.600269 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:13.599796 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:13.600269 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:13.599726 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:13.600269 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:13.599927 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z5jjd" podUID="58ec685b-55f0-42a2-9112-6fe8a813fd8a" Nov 03 21:29:13.600269 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:13.600060 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pllqd" podUID="ca003584-cd18-4740-a619-66c89853c773" Nov 03 21:29:13.611935 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:13.611702 2584 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T21:29:13.213913825Z","UUID":"2936b9e3-ae49-447d-9e92-a0c6b7e40485","Handler":null,"Name":"","Endpoint":""} Nov 03 21:29:13.618287 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:13.617888 2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 21:29:13.618287 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:13.617932 2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 21:29:13.764739 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:13.761483 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gdt8s" event={"ID":"710eeea5-8bbf-4b10-8167-45a61512ef7f","Type":"ContainerStarted","Data":"e5e26e2f574d6fd767c9fb58560f65564f52e373eb62c21551d24a22fb722053"} Nov 03 21:29:13.765111 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:13.765073 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" event={"ID":"1bc36693-102d-48e8-bb01-302b38055e05","Type":"ContainerStarted","Data":"fd1922f9ddce1819fa4782f0fa35abd5eeee40967188969ecf466cda107bd743"} Nov 03 21:29:13.813692 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:13.801336 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rwd6w" podStartSLOduration=4.81864252 podStartE2EDuration="9.801316074s" totalImagesPullingTime="4.982673554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:11.796086563 +0000 UTC m=+8.688194041" watchObservedRunningTime="2026-11-03 21:29:13.801316074 +0000 UTC m=+10.693423556" Nov 03 21:29:14.525017 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:14.524820 2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:14.525455 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:14.525435 2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:14.565269 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:14.565210 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-gdt8s" podStartSLOduration=2.820290757 podStartE2EDuration="10.565187451s" totalImagesPullingTime="7.744896694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:13.802514561 +0000 UTC m=+10.694622052" watchObservedRunningTime="2026-11-03 21:29:14.565187451 +0000 UTC m=+11.457294925" Nov 03 21:29:14.598583 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:14.598556 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:14.598758 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:14.598707 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvv5p" podUID="2a406a0d-024a-411f-94c2-953a739c214f" Nov 03 21:29:14.769862 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:14.769824 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" event={"ID":"1bc36693-102d-48e8-bb01-302b38055e05","Type":"ContainerStarted","Data":"3657fe680416365c4503ae0e6f1ba46cd1be85316445315b1094d35de4657e75"} Nov 03 21:29:14.772118 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:14.772086 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5wq87" event={"ID":"d65ba521-c50d-47b4-82d9-1910dde4fbb1","Type":"ContainerStarted","Data":"5afb0fcf56304b33ddfa6185343bb8aca7ec108012d237ce1e46043632ecafbc"} Nov 03 21:29:14.774213 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:14.774185 2584 generic.go:356] "Generic (PLEG): container finished" podID="add088ee-8843-4be8-ad89-f396481163c5" containerID="079082d63f644f8f917099442a8bfc9101ef9d9a63a74a1eac94dd3429e2d702" exitCode=0 Nov 03 21:29:14.774332 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:14.774280 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lwrg" event={"ID":"add088ee-8843-4be8-ad89-f396481163c5","Type":"ContainerDied","Data":"079082d63f644f8f917099442a8bfc9101ef9d9a63a74a1eac94dd3429e2d702"} Nov 03 21:29:14.811393 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:14.810934 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" podStartSLOduration=2.72827055 podStartE2EDuration="10.810916343s" totalImagesPullingTime="8.082645793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:29:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:14.810585657 +0000 UTC m=+11.702693146" watchObservedRunningTime="2026-11-03 21:29:14.810916343 +0000 UTC m=+11.703023823" Nov 03 21:29:14.850386 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:14.850325 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5wq87" podStartSLOduration=2.642184733 podStartE2EDuration="10.850306546s" totalImagesPullingTime="8.208121813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:14.849995784 +0000 UTC m=+11.742103265" watchObservedRunningTime="2026-11-03 21:29:14.850306546 +0000 UTC m=+11.742414026" Nov 03 21:29:15.598982 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:15.598899 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:15.598982 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:15.598945 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:15.599516 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:15.599044 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pllqd" podUID="ca003584-cd18-4740-a619-66c89853c773" Nov 03 21:29:15.599516 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:15.599184 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z5jjd" podUID="58ec685b-55f0-42a2-9112-6fe8a813fd8a" Nov 03 21:29:15.779137 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:15.778722 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rc5dj" event={"ID":"b04c787c-4de0-483a-9f96-698fa046d04b","Type":"ContainerStarted","Data":"ae3c48ea837efe30e29ada70f994468a132a5ecc0366a9a6553f35bb0d463972"} Nov 03 21:29:15.816691 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:15.816640 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rc5dj" podStartSLOduration=2.8281911060000002 podStartE2EDuration="10.816625433s" totalImagesPullingTime="7.988434327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:15.816318636 +0000 UTC m=+12.708426117" watchObservedRunningTime="2026-11-03 21:29:15.816625433 +0000 UTC m=+12.708732912" Nov 03 21:29:16.598327 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:16.598241 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:16.598475 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:16.598409 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvv5p" podUID="2a406a0d-024a-411f-94c2-953a739c214f" Nov 03 21:29:16.781977 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:16.781941 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" event={"ID":"c5013527-41c2-4355-9204-594fe21d3c8a","Type":"ContainerStarted","Data":"a7a29b675d1b2c9d4e9d8cef30ea56e50970edcbbe9a0733412f36b3488be0ed"} Nov 03 21:29:16.784601 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:16.784516 2584 generic.go:356] "Generic (PLEG): container finished" podID="add088ee-8843-4be8-ad89-f396481163c5" containerID="b2b4cf4d77b10dfacee0863c3017dc40d76cf0b33ebde177673c01eb7ac6628a" exitCode=0 Nov 03 21:29:16.784601 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:16.784550 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lwrg" event={"ID":"add088ee-8843-4be8-ad89-f396481163c5","Type":"ContainerDied","Data":"b2b4cf4d77b10dfacee0863c3017dc40d76cf0b33ebde177673c01eb7ac6628a"} Nov 03 21:29:16.818559 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:16.818521 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" podStartSLOduration=2.480324688 podStartE2EDuration="12.818509248s" totalImagesPullingTime="10.33818456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:16.818244798 +0000 UTC m=+13.710352274" watchObservedRunningTime="2026-11-03 21:29:16.818509248 +0000 UTC m=+13.710616749" Nov 03 21:29:17.598801 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:17.598769 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:17.598933 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:17.598905 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pllqd" podUID="ca003584-cd18-4740-a619-66c89853c773" Nov 03 21:29:17.599415 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:17.599272 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:17.599415 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:17.599372 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z5jjd" podUID="58ec685b-55f0-42a2-9112-6fe8a813fd8a" Nov 03 21:29:17.789431 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:17.789394 2584 generic.go:356] "Generic (PLEG): container finished" podID="add088ee-8843-4be8-ad89-f396481163c5" containerID="47c5c03cbbf78059cb71d976ed066ca70fab06083d088f691a2c7ed58a0599c8" exitCode=0 Nov 03 21:29:17.789995 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:17.789486 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lwrg" event={"ID":"add088ee-8843-4be8-ad89-f396481163c5","Type":"ContainerDied","Data":"47c5c03cbbf78059cb71d976ed066ca70fab06083d088f691a2c7ed58a0599c8"} Nov 03 21:29:18.599055 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:18.598797 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:18.599227 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:18.599163 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvv5p" podUID="2a406a0d-024a-411f-94c2-953a739c214f" Nov 03 21:29:19.598968 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:19.598933 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:19.599443 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:19.598937 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:19.599443 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:19.599076 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z5jjd" podUID="58ec685b-55f0-42a2-9112-6fe8a813fd8a" Nov 03 21:29:19.599443 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:19.599141 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pllqd" podUID="ca003584-cd18-4740-a619-66c89853c773" Nov 03 21:29:19.796557 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:19.796523 2584 generic.go:356] "Generic (PLEG): container finished" podID="add088ee-8843-4be8-ad89-f396481163c5" containerID="13ba642705953d82f39c75af97844e8ff9b5dafba7af951d60642f8b91fd5e76" exitCode=0 Nov 03 21:29:19.796732 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:19.796582 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lwrg" event={"ID":"add088ee-8843-4be8-ad89-f396481163c5","Type":"ContainerDied","Data":"13ba642705953d82f39c75af97844e8ff9b5dafba7af951d60642f8b91fd5e76"} Nov 03 21:29:20.599029 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:20.599003 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:20.599395 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:20.599100 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvv5p" podUID="2a406a0d-024a-411f-94c2-953a739c214f" Nov 03 21:29:20.800902 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:20.800874 2584 generic.go:356] "Generic (PLEG): container finished" podID="add088ee-8843-4be8-ad89-f396481163c5" containerID="db0dc9577da37c57b1a273822b0821bbbb08c018983f1360f4cb9bef90293490" exitCode=0 Nov 03 21:29:20.800977 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:20.800946 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lwrg" event={"ID":"add088ee-8843-4be8-ad89-f396481163c5","Type":"ContainerDied","Data":"db0dc9577da37c57b1a273822b0821bbbb08c018983f1360f4cb9bef90293490"} Nov 03 21:29:20.802780 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:20.802758 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" event={"ID":"0d3ddfc6-6d13-4b06-983f-75cca78543ea","Type":"ContainerStarted","Data":"4f32074fa8e617f5c0de49d11b4f1bef5d8bdb08a30c0fbaebba01737cfdcce4"} Nov 03 21:29:20.802865 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:20.802817 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" event={"ID":"0d3ddfc6-6d13-4b06-983f-75cca78543ea","Type":"ContainerStarted","Data":"54d324331e119bbe132fc34d8b7ae8dce4683cfbf9d712c649cfb85b0e3208d7"} Nov 03 21:29:21.387757 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:21.387543 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs\") pod \"network-metrics-daemon-qvv5p\" (UID: \"2a406a0d-024a-411f-94c2-953a739c214f\") " pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:21.387953 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:21.387702 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:29:21.387953 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:21.387848 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs podName:2a406a0d-024a-411f-94c2-953a739c214f nodeName:}" failed. No retries permitted until 2026-11-03 21:29:37.387834262 +0000 UTC m=+34.279941716 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs") pod "network-metrics-daemon-qvv5p" (UID: "2a406a0d-024a-411f-94c2-953a739c214f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 21:29:21.488193 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:21.488149 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:21.488339 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:21.488219 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dl8\" (UniqueName: \"kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8\") pod \"network-check-target-pllqd\" (UID: \"ca003584-cd18-4740-a619-66c89853c773\") " pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:21.488339 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:21.488293 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 21:29:21.488339 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:21.488309 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 21:29:21.488339 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:21.488323 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 21:29:21.488339 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:21.488332 2584 projected.go:196] Error preparing data for projected volume kube-api-access-d7dl8 for pod openshift-network-diagnostics/network-check-target-pllqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:29:21.488504 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:21.488355 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret podName:58ec685b-55f0-42a2-9112-6fe8a813fd8a nodeName:}" failed. No retries permitted until 2026-11-03 21:29:37.488340263 +0000 UTC m=+34.380447741 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret") pod "global-pull-secret-syncer-z5jjd" (UID: "58ec685b-55f0-42a2-9112-6fe8a813fd8a") : object "kube-system"/"original-pull-secret" not registered Nov 03 21:29:21.488504 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:21.488370 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8 podName:ca003584-cd18-4740-a619-66c89853c773 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:37.488364844 +0000 UTC m=+34.380472299 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7dl8" (UniqueName: "kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8") pod "network-check-target-pllqd" (UID: "ca003584-cd18-4740-a619-66c89853c773") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 21:29:21.599314 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:21.599283 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:21.599314 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:21.599318 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:21.599717 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:21.599394 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z5jjd" podUID="58ec685b-55f0-42a2-9112-6fe8a813fd8a" Nov 03 21:29:21.599717 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:21.599444 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pllqd" podUID="ca003584-cd18-4740-a619-66c89853c773" Nov 03 21:29:21.808176 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:21.808140 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lwrg" event={"ID":"add088ee-8843-4be8-ad89-f396481163c5","Type":"ContainerStarted","Data":"6c278331fdb03e09a904b5a8ef33c3d296fbd5cb425b527f33480397d8b0bc5c"} Nov 03 21:29:21.810883 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:21.810859 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" event={"ID":"0d3ddfc6-6d13-4b06-983f-75cca78543ea","Type":"ContainerStarted","Data":"57accb698e1d63061dedd50a9e1fda8297d8305a74cd9110f8f41b1b14de24a9"} Nov 03 21:29:21.810998 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:21.810913 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" event={"ID":"0d3ddfc6-6d13-4b06-983f-75cca78543ea","Type":"ContainerStarted","Data":"4f6b927c2e21d813327c2eaea7f90c92d28314348ccb45a1b79ca82d010f22be"} Nov 03 21:29:21.810998 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:21.810953 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" event={"ID":"0d3ddfc6-6d13-4b06-983f-75cca78543ea","Type":"ContainerStarted","Data":"edbc19c95344d559cf486697aa35d5e66cacf0270afe82bfd0403862447e0bbe"} Nov 03 21:29:21.810998 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:21.810979 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" event={"ID":"0d3ddfc6-6d13-4b06-983f-75cca78543ea","Type":"ContainerStarted","Data":"12bbf7b4a4cb4297ffa944029588aba7a35ac2d394915c181839dd75277885ee"} Nov 03 21:29:21.853566 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:21.852873 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2lwrg" podStartSLOduration=7.346631281 podStartE2EDuration="17.852851402s" totalImagesPullingTime="10.506220121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:29:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:21.851091205 +0000 UTC m=+18.743198682" watchObservedRunningTime="2026-11-03 21:29:21.852851402 +0000 UTC m=+18.744958879" Nov 03 21:29:22.456774 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:22.456741 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:22.456942 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:22.456883 2584 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 21:29:22.457312 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:22.457298 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:22.598439 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:22.598401 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:22.598593 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:22.598510 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvv5p" podUID="2a406a0d-024a-411f-94c2-953a739c214f" Nov 03 21:29:23.599112 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:23.599070 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:23.599648 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:23.599208 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pllqd" podUID="ca003584-cd18-4740-a619-66c89853c773" Nov 03 21:29:23.599648 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:23.599237 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:23.599648 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:23.599302 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z5jjd" podUID="58ec685b-55f0-42a2-9112-6fe8a813fd8a" Nov 03 21:29:23.817732 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:23.817698 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" event={"ID":"0d3ddfc6-6d13-4b06-983f-75cca78543ea","Type":"ContainerStarted","Data":"5ec6a0733896796a86353e636b077dac84fbe583e224d0ae275f536efeaeb29b"} Nov 03 21:29:24.598384 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:24.598354 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:24.598548 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:24.598468 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvv5p" podUID="2a406a0d-024a-411f-94c2-953a739c214f" Nov 03 21:29:25.598848 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:25.598817 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:25.598848 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:25.598840 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:25.599521 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:25.598920 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pllqd" podUID="ca003584-cd18-4740-a619-66c89853c773" Nov 03 21:29:25.599521 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:25.599023 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z5jjd" podUID="58ec685b-55f0-42a2-9112-6fe8a813fd8a" Nov 03 21:29:25.826555 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:25.826521 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" event={"ID":"0d3ddfc6-6d13-4b06-983f-75cca78543ea","Type":"ContainerStarted","Data":"cf96e1d4259a1e3ef2a713d6c3808ccd8ee0db8d3a132217dde5151474d6f31d"} Nov 03 21:29:25.826800 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:25.826784 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:25.841670 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:25.841646 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:25.905030 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:25.904935 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" podStartSLOduration=7.159084146 podStartE2EDuration="21.904923325s" totalImagesPullingTime="14.745839179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:25.904446672 +0000 UTC m=+22.796554147" watchObservedRunningTime="2026-11-03 21:29:25.904923325 +0000 UTC m=+22.797030831" Nov 03 21:29:26.598836 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:26.598811 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:26.598999 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:26.598912 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvv5p" podUID="2a406a0d-024a-411f-94c2-953a739c214f" Nov 03 21:29:26.829945 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:26.829830 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:26.829945 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:26.829906 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:26.844533 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:26.844508 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:27.197449 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:27.197413 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z5jjd"] Nov 03 21:29:27.197921 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:27.197901 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:27.198189 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:27.198159 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z5jjd" podUID="58ec685b-55f0-42a2-9112-6fe8a813fd8a" Nov 03 21:29:27.200957 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:27.200933 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pllqd"] Nov 03 21:29:27.201072 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:27.201025 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:27.201143 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:27.201119 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pllqd" podUID="ca003584-cd18-4740-a619-66c89853c773" Nov 03 21:29:27.201701 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:27.201685 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvv5p"] Nov 03 21:29:27.201775 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:27.201765 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:27.201882 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:27.201864 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvv5p" podUID="2a406a0d-024a-411f-94c2-953a739c214f" Nov 03 21:29:28.598879 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:28.598837 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:28.598879 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:28.598889 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:28.599286 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:28.598960 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvv5p" podUID="2a406a0d-024a-411f-94c2-953a739c214f" Nov 03 21:29:28.599286 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:28.599016 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pllqd" podUID="ca003584-cd18-4740-a619-66c89853c773" Nov 03 21:29:29.599108 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:29.599077 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:29.599610 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:29.599532 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z5jjd" podUID="58ec685b-55f0-42a2-9112-6fe8a813fd8a" Nov 03 21:29:30.598924 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:30.598900 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:30.599112 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:30.598900 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:30.599112 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:30.598993 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pllqd" podUID="ca003584-cd18-4740-a619-66c89853c773" Nov 03 21:29:30.599112 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:30.599074 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvv5p" podUID="2a406a0d-024a-411f-94c2-953a739c214f" Nov 03 21:29:31.598479 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:31.598400 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:31.598606 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:31.598511 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z5jjd" podUID="58ec685b-55f0-42a2-9112-6fe8a813fd8a" Nov 03 21:29:32.598402 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:32.598378 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:32.598802 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:32.598462 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pllqd" podUID="ca003584-cd18-4740-a619-66c89853c773" Nov 03 21:29:32.598802 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:32.598378 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:32.598802 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:32.598550 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvv5p" podUID="2a406a0d-024a-411f-94c2-953a739c214f" Nov 03 21:29:33.373867 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.373792 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeReady" Nov 03 21:29:33.374000 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.373931 2584 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 21:29:33.417027 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.416996 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pp685"] Nov 03 21:29:33.424868 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.424844 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7f6xj"] Nov 03 21:29:33.425003 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.424952 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pp685" Nov 03 21:29:33.427854 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.427827 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mq2cr"] Nov 03 21:29:33.428762 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.428171 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7f6xj" Nov 03 21:29:33.431744 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.431726 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr"] Nov 03 21:29:33.431868 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.431852 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mq2cr" Nov 03 21:29:33.434688 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.434656 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-9f5c5b5d5-hpzcq"] Nov 03 21:29:33.434763 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.434723 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:33.438115 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.438096 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fd7fc569c-h759l"] Nov 03 21:29:33.438210 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.438157 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:33.440953 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.440933 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wxk7n"] Nov 03 21:29:33.441728 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.441097 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.443849 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.443827 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zthx8"] Nov 03 21:29:33.444078 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.444064 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.450121 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.450101 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 21:29:33.450365 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.450349 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 21:29:33.456581 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.456557 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb"] Nov 03 21:29:33.457354 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.457337 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zthx8" Nov 03 21:29:33.459180 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.459133 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 21:29:33.459273 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.459190 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 21:29:33.459273 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.459255 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 21:29:33.459749 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.459341 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 21:29:33.460491 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.460469 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 21:29:33.462199 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.462179 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk"] Nov 03 21:29:33.462302 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.462284 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:33.465606 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.465589 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f"] Nov 03 21:29:33.465724 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.465706 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:33.470418 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.470373 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-w682j"] Nov 03 21:29:33.470754 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.470735 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:33.474615 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.474396 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ltzz4"] Nov 03 21:29:33.474615 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.474567 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:33.478310 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.478291 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bf564b6d-c6gsv"] Nov 03 21:29:33.479140 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.479123 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:33.484182 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.484164 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-m2hhx"] Nov 03 21:29:33.484677 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.484492 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.489784 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.489764 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7t4b8"] Nov 03 21:29:33.489911 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.489885 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m2hhx" Nov 03 21:29:33.497346 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.497328 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x4wh2"] Nov 03 21:29:33.497539 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.497522 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:33.503392 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503374 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pp685"] Nov 03 21:29:33.503487 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503409 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7f6xj"] Nov 03 21:29:33.503487 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503432 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mq2cr"] Nov 03 21:29:33.503487 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503434 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x4wh2" Nov 03 21:29:33.503487 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503452 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zthx8"] Nov 03 21:29:33.503487 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503474 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb"] Nov 03 21:29:33.503736 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503492 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wxk7n"] Nov 03 21:29:33.503736 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503511 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-9f5c5b5d5-hpzcq"] Nov 03 21:29:33.503736 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503533 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fd7fc569c-h759l"] Nov 03 21:29:33.503736 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503552 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk"] Nov 03 21:29:33.503736 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503575 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7t4b8"] Nov 03 21:29:33.503736 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503600 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr"] Nov 03 21:29:33.503736 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503620 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ltzz4"] Nov 03 21:29:33.503736 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503642 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x4wh2"] Nov 03 21:29:33.503736 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503676 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-m2hhx"] Nov 03 21:29:33.503736 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503698 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-w682j"] Nov 03 21:29:33.503736 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503719 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bf564b6d-c6gsv"] Nov 03 21:29:33.503736 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.503737 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f"] Nov 03 21:29:33.544954 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.544932 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:29:33.545035 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.544943 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 21:29:33.545229 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.545211 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 21:29:33.550368 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.545303 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 21:29:33.550368 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.549985 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 21:29:33.550368 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.545391 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 21:29:33.550368 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.547228 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 21:29:33.550368 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.547405 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 21:29:33.550368 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.550216 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 21:29:33.550368 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.547637 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 21:29:33.550368 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.550317 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 21:29:33.550863 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.547899 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:29:33.550863 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.550518 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 21:29:33.550863 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.548385 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 21:29:33.550863 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.548129 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 21:29:33.550863 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.548632 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 21:29:33.550863 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.550678 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 21:29:33.550863 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.546864 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:29:33.550863 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.547778 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 21:29:33.550863 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.549155 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:29:33.550863 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.549348 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 21:29:33.550863 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.549465 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 21:29:33.550863 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.549421 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:29:33.550863 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.549548 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 21:29:33.550863 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.548122 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 21:29:33.551420 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.548503 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 21:29:33.551420 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.551066 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 21:29:33.557632 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.557618 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 21:29:33.558489 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.558477 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 21:29:33.570974 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.570951 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 21:29:33.570974 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.570964 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 21:29:33.571208 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.571188 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 21:29:33.571295 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.571281 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 21:29:33.571295 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.571294 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 21:29:33.571415 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.571301 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 21:29:33.571415 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.571315 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 21:29:33.571415 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.571318 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 21:29:33.571415 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.571342 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-qmg6s\"" Nov 03 21:29:33.571415 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.571353 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 21:29:33.571415 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.571359 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qc6g7\"" Nov 03 21:29:33.571415 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.571296 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 21:29:33.571415 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.571370 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 21:29:33.571415 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.571397 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fv5g9\"" Nov 03 21:29:33.571645 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.571467 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 21:29:33.571645 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.571503 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 21:29:33.571645 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.571579 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x8jbx\"" Nov 03 21:29:33.575797 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.575772 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ddbb8fca-c02a-4e5f-a31f-919194f46f12-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:33.575797 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.575796 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfcx5\" (UniqueName: \"kubernetes.io/projected/0852ff0f-d078-452f-bb56-3c13512f57ce-kube-api-access-xfcx5\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:33.575933 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.575813 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-image-registry-private-configuration\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.575933 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.575829 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-certificates\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.575933 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.575851 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmdxx\" (UniqueName: \"kubernetes.io/projected/37984c5f-7e58-4323-81f0-4bc7d485ab88-kube-api-access-bmdxx\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.575933 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.575909 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct582\" (UniqueName: \"kubernetes.io/projected/9f173fd7-1ddb-4616-91b2-35a2bd6e0341-kube-api-access-ct582\") pod \"network-check-source-6f78d59f99-m2hhx\" (UID: \"9f173fd7-1ddb-4616-91b2-35a2bd6e0341\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m2hhx" Nov 03 21:29:33.576058 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.575981 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btpnr\" (UniqueName: \"kubernetes.io/projected/efc25e42-1fb2-4fb4-b991-ce83b2d174d1-kube-api-access-btpnr\") pod \"kas-connection-checker-685895c7c9-pp685\" (UID: \"efc25e42-1fb2-4fb4-b991-ce83b2d174d1\") " pod="kube-system/kas-connection-checker-685895c7c9-pp685" Nov 03 21:29:33.576058 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576001 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37984c5f-7e58-4323-81f0-4bc7d485ab88-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.576058 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576025 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh5pc\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-kube-api-access-mh5pc\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.576058 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576042 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37a8184c-e9f5-4d96-a816-e0191c32ed06-config\") pod \"service-ca-operator-544d749c8c-r9qwk\" (UID: \"37a8184c-e9f5-4d96-a816-e0191c32ed06\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:33.576166 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576106 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mddx\" (UniqueName: \"kubernetes.io/projected/37a8184c-e9f5-4d96-a816-e0191c32ed06-kube-api-access-6mddx\") pod \"service-ca-operator-544d749c8c-r9qwk\" (UID: \"37a8184c-e9f5-4d96-a816-e0191c32ed06\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:33.576166 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576129 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:33.576166 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576155 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sldfp\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-kube-api-access-sldfp\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.576245 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576197 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9205351c-9ed0-4a10-8c52-0124c39a20fe-nginx-conf\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:33.576274 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576241 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-installation-pull-secrets\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.576274 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576260 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8ssv\" (UniqueName: \"kubernetes.io/projected/cb8cd0fc-653b-4f07-a501-f121ec73b7d5-kube-api-access-k8ssv\") pod \"volume-data-source-validator-58c9f5f46d-zthx8\" (UID: \"cb8cd0fc-653b-4f07-a501-f121ec73b7d5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zthx8" Nov 03 21:29:33.576332 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576319 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd1064c9-bade-4061-9b89-5d15e6a26a3d-ca-trust-extracted\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.576366 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576347 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37984c5f-7e58-4323-81f0-4bc7d485ab88-serving-cert\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.576366 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576361 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:33.576419 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576376 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nc6h\" (UniqueName: \"kubernetes.io/projected/a734caac-db4c-4f6e-8dd0-415843ecd1b8-kube-api-access-7nc6h\") pod \"kas-connection-checker-685895c7c9-mq2cr\" (UID: \"a734caac-db4c-4f6e-8dd0-415843ecd1b8\") " pod="kube-system/kas-connection-checker-685895c7c9-mq2cr" Nov 03 21:29:33.576447 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576425 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37984c5f-7e58-4323-81f0-4bc7d485ab88-tmp\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.576487 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576444 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-stats-auth\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:33.576487 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576474 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6467adf-c26e-46ae-a212-98c9846bfe0e-ca-trust-extracted\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.576544 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576512 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/37984c5f-7e58-4323-81f0-4bc7d485ab88-snapshots\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.576544 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576534 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-config-volume\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:33.576601 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576575 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-trusted-ca\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:33.576628 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576607 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37984c5f-7e58-4323-81f0-4bc7d485ab88-service-ca-bundle\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.576678 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576629 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-bound-sa-token\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.576678 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576645 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9s65\" (UniqueName: \"kubernetes.io/projected/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2-kube-api-access-b9s65\") pod \"ingress-canary-x4wh2\" (UID: \"2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2\") " pod="openshift-ingress-canary/ingress-canary-x4wh2" Nov 03 21:29:33.576743 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576678 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-installation-pull-secrets\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.576743 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576730 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx68k\" (UniqueName: \"kubernetes.io/projected/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-kube-api-access-kx68k\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:33.576803 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576746 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-trusted-ca\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.576803 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576767 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab012cc6-ec86-41af-b6a1-e9278469c6de-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hplgb\" (UID: \"ab012cc6-ec86-41af-b6a1-e9278469c6de\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:33.576859 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576818 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:33.576859 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576841 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-bound-sa-token\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.576911 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576859 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:33.576911 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576881 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-config\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:33.576911 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576903 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbkft\" (UniqueName: \"kubernetes.io/projected/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-kube-api-access-jbkft\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:33.576993 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576927 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-default-certificate\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:33.576993 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576962 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2-cert\") pod \"ingress-canary-x4wh2\" (UID: \"2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2\") " pod="openshift-ingress-canary/ingress-canary-x4wh2" Nov 03 21:29:33.576993 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.576987 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37a8184c-e9f5-4d96-a816-e0191c32ed06-serving-cert\") pod \"service-ca-operator-544d749c8c-r9qwk\" (UID: \"37a8184c-e9f5-4d96-a816-e0191c32ed06\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:33.577068 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.577036 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkjh7\" (UniqueName: \"kubernetes.io/projected/ddbb8fca-c02a-4e5f-a31f-919194f46f12-kube-api-access-fkjh7\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:33.577068 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.577061 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:33.577120 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.577082 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-tmp-dir\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:33.577120 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.577102 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pwnr\" (UniqueName: \"kubernetes.io/projected/57dae9a4-2abc-4829-9beb-f61383f46bfe-kube-api-access-8pwnr\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:33.577177 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.577123 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmt5p\" (UniqueName: \"kubernetes.io/projected/ab012cc6-ec86-41af-b6a1-e9278469c6de-kube-api-access-mmt5p\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hplgb\" (UID: \"ab012cc6-ec86-41af-b6a1-e9278469c6de\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:33.577177 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.577144 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:33.577232 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.577174 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab012cc6-ec86-41af-b6a1-e9278469c6de-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hplgb\" (UID: \"ab012cc6-ec86-41af-b6a1-e9278469c6de\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:33.577232 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.577194 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-trusted-ca\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.577232 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.577213 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-image-registry-private-configuration\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.577232 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.577230 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-serving-cert\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:33.577340 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.577244 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.577340 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.577264 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:33.577340 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.577313 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-certificates\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.577424 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.577336 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz5g7\" (UniqueName: \"kubernetes.io/projected/d8a307b9-8e43-4b73-941d-d5e2c035da14-kube-api-access-dz5g7\") pod \"kas-connection-checker-685895c7c9-7f6xj\" (UID: \"d8a307b9-8e43-4b73-941d-d5e2c035da14\") " pod="kube-system/kas-connection-checker-685895c7c9-7f6xj" Nov 03 21:29:33.577424 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.577365 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.598266 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.598244 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:33.658357 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.658302 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 21:29:33.658357 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.658347 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rf9jq\"" Nov 03 21:29:33.678227 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.678199 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/37984c5f-7e58-4323-81f0-4bc7d485ab88-snapshots\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.678337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.678235 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-config-volume\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:33.678337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.678252 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-trusted-ca\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:33.678337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.678267 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37984c5f-7e58-4323-81f0-4bc7d485ab88-service-ca-bundle\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.678337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.678288 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-bound-sa-token\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.678337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.678315 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9s65\" (UniqueName: \"kubernetes.io/projected/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2-kube-api-access-b9s65\") pod \"ingress-canary-x4wh2\" (UID: \"2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2\") " pod="openshift-ingress-canary/ingress-canary-x4wh2" Nov 03 21:29:33.678337 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.678341 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-installation-pull-secrets\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.678652 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.678364 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx68k\" (UniqueName: \"kubernetes.io/projected/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-kube-api-access-kx68k\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:33.678652 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.678387 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-trusted-ca\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.678652 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.678575 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab012cc6-ec86-41af-b6a1-e9278469c6de-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hplgb\" (UID: \"ab012cc6-ec86-41af-b6a1-e9278469c6de\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:33.678869 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.678848 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/37984c5f-7e58-4323-81f0-4bc7d485ab88-snapshots\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.678968 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.678946 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37984c5f-7e58-4323-81f0-4bc7d485ab88-service-ca-bundle\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.679052 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679033 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:33.679111 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679067 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-bound-sa-token\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.679157 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679097 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:33.679157 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679148 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-config\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:33.679249 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679150 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-config-volume\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:33.679249 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.679209 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 21:29:33.679249 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679213 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-trusted-ca\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:33.679382 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.679269 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert podName:9205351c-9ed0-4a10-8c52-0124c39a20fe nodeName:}" failed. No retries permitted until 2026-11-03 21:29:34.179256861 +0000 UTC m=+31.071364319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w682j" (UID: "9205351c-9ed0-4a10-8c52-0124c39a20fe") : secret "networking-console-plugin-cert" not found Nov 03 21:29:33.679382 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.679299 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:33.679382 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679341 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbkft\" (UniqueName: \"kubernetes.io/projected/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-kube-api-access-jbkft\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:33.679382 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.679362 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls podName:ddbb8fca-c02a-4e5f-a31f-919194f46f12 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:34.179345793 +0000 UTC m=+31.071453267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wq5rr" (UID: "ddbb8fca-c02a-4e5f-a31f-919194f46f12") : secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:33.679573 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679396 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-default-certificate\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:33.679573 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679427 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2-cert\") pod \"ingress-canary-x4wh2\" (UID: \"2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2\") " pod="openshift-ingress-canary/ingress-canary-x4wh2" Nov 03 21:29:33.679573 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679452 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37a8184c-e9f5-4d96-a816-e0191c32ed06-serving-cert\") pod \"service-ca-operator-544d749c8c-r9qwk\" (UID: \"37a8184c-e9f5-4d96-a816-e0191c32ed06\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:33.679573 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679471 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-trusted-ca\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.679573 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679484 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjh7\" (UniqueName: \"kubernetes.io/projected/ddbb8fca-c02a-4e5f-a31f-919194f46f12-kube-api-access-fkjh7\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:33.679573 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679526 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:33.679573 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679558 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-tmp-dir\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:33.679908 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679586 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pwnr\" (UniqueName: \"kubernetes.io/projected/57dae9a4-2abc-4829-9beb-f61383f46bfe-kube-api-access-8pwnr\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:33.679908 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679616 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmt5p\" (UniqueName: \"kubernetes.io/projected/ab012cc6-ec86-41af-b6a1-e9278469c6de-kube-api-access-mmt5p\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hplgb\" (UID: \"ab012cc6-ec86-41af-b6a1-e9278469c6de\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:33.679908 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679646 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:33.679908 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679691 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab012cc6-ec86-41af-b6a1-e9278469c6de-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hplgb\" (UID: \"ab012cc6-ec86-41af-b6a1-e9278469c6de\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:33.679908 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679716 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-trusted-ca\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.679908 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679741 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-image-registry-private-configuration\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.679908 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679771 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-serving-cert\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:33.679908 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679777 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-config\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:33.679908 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679795 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.679908 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679817 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:33.679908 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679840 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-certificates\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.679908 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.679859 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 21:29:33.679908 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.679877 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz5g7\" (UniqueName: \"kubernetes.io/projected/d8a307b9-8e43-4b73-941d-d5e2c035da14-kube-api-access-dz5g7\") pod \"kas-connection-checker-685895c7c9-7f6xj\" (UID: \"d8a307b9-8e43-4b73-941d-d5e2c035da14\") " pod="kube-system/kas-connection-checker-685895c7c9-7f6xj" Nov 03 21:29:33.679908 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.679897 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs podName:0852ff0f-d078-452f-bb56-3c13512f57ce nodeName:}" failed. No retries permitted until 2026-11-03 21:29:34.179884183 +0000 UTC m=+31.071991654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs") pod "router-default-9f5c5b5d5-hpzcq" (UID: "0852ff0f-d078-452f-bb56-3c13512f57ce") : secret "router-metrics-certs-default" not found Nov 03 21:29:33.680582 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.680124 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-tmp-dir\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:33.680582 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.680237 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:33.680582 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.680534 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab012cc6-ec86-41af-b6a1-e9278469c6de-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hplgb\" (UID: \"ab012cc6-ec86-41af-b6a1-e9278469c6de\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:33.680783 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.680629 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:33.680783 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.680643 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd7fc569c-h759l: secret "image-registry-tls" not found Nov 03 21:29:33.680783 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.680700 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls podName:cd1064c9-bade-4061-9b89-5d15e6a26a3d nodeName:}" failed. No retries permitted until 2026-11-03 21:29:34.180687668 +0000 UTC m=+31.072795133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls") pod "image-registry-7fd7fc569c-h759l" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d") : secret "image-registry-tls" not found Nov 03 21:29:33.682651 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.681393 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-trusted-ca\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.682651 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.681488 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 21:29:33.682651 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.681541 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls podName:2a8bade0-e7b8-4d61-b71f-fdb4b2517320 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:34.181526871 +0000 UTC m=+31.073634340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls") pod "dns-default-7t4b8" (UID: "2a8bade0-e7b8-4d61-b71f-fdb4b2517320") : secret "dns-default-metrics-tls" not found Nov 03 21:29:33.682651 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682011 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-certificates\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.682651 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682053 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.682651 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682082 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ddbb8fca-c02a-4e5f-a31f-919194f46f12-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:33.682651 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682108 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfcx5\" (UniqueName: \"kubernetes.io/projected/0852ff0f-d078-452f-bb56-3c13512f57ce-kube-api-access-xfcx5\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:33.682651 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682134 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-image-registry-private-configuration\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.682651 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682157 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-certificates\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.682651 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682187 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmdxx\" (UniqueName: \"kubernetes.io/projected/37984c5f-7e58-4323-81f0-4bc7d485ab88-kube-api-access-bmdxx\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.682651 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682216 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct582\" (UniqueName: \"kubernetes.io/projected/9f173fd7-1ddb-4616-91b2-35a2bd6e0341-kube-api-access-ct582\") pod \"network-check-source-6f78d59f99-m2hhx\" (UID: \"9f173fd7-1ddb-4616-91b2-35a2bd6e0341\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m2hhx" Nov 03 21:29:33.682651 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682267 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btpnr\" (UniqueName: \"kubernetes.io/projected/efc25e42-1fb2-4fb4-b991-ce83b2d174d1-kube-api-access-btpnr\") pod \"kas-connection-checker-685895c7c9-pp685\" (UID: \"efc25e42-1fb2-4fb4-b991-ce83b2d174d1\") " pod="kube-system/kas-connection-checker-685895c7c9-pp685" Nov 03 21:29:33.682651 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682292 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37984c5f-7e58-4323-81f0-4bc7d485ab88-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.682651 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682316 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh5pc\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-kube-api-access-mh5pc\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.682651 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682341 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37a8184c-e9f5-4d96-a816-e0191c32ed06-config\") pod \"service-ca-operator-544d749c8c-r9qwk\" (UID: \"37a8184c-e9f5-4d96-a816-e0191c32ed06\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:33.682651 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682389 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mddx\" (UniqueName: \"kubernetes.io/projected/37a8184c-e9f5-4d96-a816-e0191c32ed06-kube-api-access-6mddx\") pod \"service-ca-operator-544d749c8c-r9qwk\" (UID: \"37a8184c-e9f5-4d96-a816-e0191c32ed06\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:33.683394 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682415 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:33.683394 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682440 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sldfp\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-kube-api-access-sldfp\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.683394 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682471 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9205351c-9ed0-4a10-8c52-0124c39a20fe-nginx-conf\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:33.683394 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682505 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-installation-pull-secrets\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.683394 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682534 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ssv\" (UniqueName: \"kubernetes.io/projected/cb8cd0fc-653b-4f07-a501-f121ec73b7d5-kube-api-access-k8ssv\") pod \"volume-data-source-validator-58c9f5f46d-zthx8\" (UID: \"cb8cd0fc-653b-4f07-a501-f121ec73b7d5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zthx8" Nov 03 21:29:33.683394 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682566 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd1064c9-bade-4061-9b89-5d15e6a26a3d-ca-trust-extracted\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.683394 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.683064 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:33.683394 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.683083 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf564b6d-c6gsv: secret "image-registry-tls" not found Nov 03 21:29:33.683394 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.683139 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls podName:a6467adf-c26e-46ae-a212-98c9846bfe0e nodeName:}" failed. No retries permitted until 2026-11-03 21:29:34.183123361 +0000 UTC m=+31.075230836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls") pod "image-registry-7bf564b6d-c6gsv" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e") : secret "image-registry-tls" not found Nov 03 21:29:33.683829 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.683425 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2-cert\") pod \"ingress-canary-x4wh2\" (UID: \"2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2\") " pod="openshift-ingress-canary/ingress-canary-x4wh2" Nov 03 21:29:33.683829 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.682607 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37984c5f-7e58-4323-81f0-4bc7d485ab88-serving-cert\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.683829 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.683492 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:33.683829 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.683527 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nc6h\" (UniqueName: \"kubernetes.io/projected/a734caac-db4c-4f6e-8dd0-415843ecd1b8-kube-api-access-7nc6h\") pod \"kas-connection-checker-685895c7c9-mq2cr\" (UID: \"a734caac-db4c-4f6e-8dd0-415843ecd1b8\") " pod="kube-system/kas-connection-checker-685895c7c9-mq2cr" Nov 03 21:29:33.683829 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.683562 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37984c5f-7e58-4323-81f0-4bc7d485ab88-tmp\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.683829 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.683587 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-stats-auth\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:33.683829 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.683813 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6467adf-c26e-46ae-a212-98c9846bfe0e-ca-trust-extracted\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.684213 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.684155 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ddbb8fca-c02a-4e5f-a31f-919194f46f12-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:33.684213 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.684166 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6467adf-c26e-46ae-a212-98c9846bfe0e-ca-trust-extracted\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.684349 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.684325 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37984c5f-7e58-4323-81f0-4bc7d485ab88-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.684538 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.684517 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37a8184c-e9f5-4d96-a816-e0191c32ed06-serving-cert\") pod \"service-ca-operator-544d749c8c-r9qwk\" (UID: \"37a8184c-e9f5-4d96-a816-e0191c32ed06\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:33.684716 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.684647 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-certificates\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.684780 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.684746 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab012cc6-ec86-41af-b6a1-e9278469c6de-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hplgb\" (UID: \"ab012cc6-ec86-41af-b6a1-e9278469c6de\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:33.687444 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.685130 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-installation-pull-secrets\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.687444 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.685869 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle podName:0852ff0f-d078-452f-bb56-3c13512f57ce nodeName:}" failed. No retries permitted until 2026-11-03 21:29:34.185853476 +0000 UTC m=+31.077960952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle") pod "router-default-9f5c5b5d5-hpzcq" (UID: "0852ff0f-d078-452f-bb56-3c13512f57ce") : configmap references non-existent config key: service-ca.crt Nov 03 21:29:33.687444 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.685878 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 21:29:33.687444 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.686139 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37a8184c-e9f5-4d96-a816-e0191c32ed06-config\") pod \"service-ca-operator-544d749c8c-r9qwk\" (UID: \"37a8184c-e9f5-4d96-a816-e0191c32ed06\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:33.687444 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.686630 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-image-registry-private-configuration\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.687444 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.686633 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37984c5f-7e58-4323-81f0-4bc7d485ab88-tmp\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.687444 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.687343 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd1064c9-bade-4061-9b89-5d15e6a26a3d-ca-trust-extracted\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.687444 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:33.685933 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls podName:57dae9a4-2abc-4829-9beb-f61383f46bfe nodeName:}" failed. No retries permitted until 2026-11-03 21:29:34.185918572 +0000 UTC m=+31.078026043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6fc7f" (UID: "57dae9a4-2abc-4829-9beb-f61383f46bfe") : secret "samples-operator-tls" not found Nov 03 21:29:33.690184 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.687623 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9205351c-9ed0-4a10-8c52-0124c39a20fe-nginx-conf\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:33.690184 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.688079 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-default-certificate\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:33.690184 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.688087 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-serving-cert\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:33.690274 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.690195 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-installation-pull-secrets\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.690970 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.690951 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37984c5f-7e58-4323-81f0-4bc7d485ab88-serving-cert\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.691091 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.691070 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-image-registry-private-configuration\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.692103 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.692086 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-stats-auth\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:33.696901 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.696879 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-bound-sa-token\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.696901 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.696893 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx68k\" (UniqueName: \"kubernetes.io/projected/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-kube-api-access-kx68k\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:33.697075 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.697056 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9s65\" (UniqueName: \"kubernetes.io/projected/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2-kube-api-access-b9s65\") pod \"ingress-canary-x4wh2\" (UID: \"2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2\") " pod="openshift-ingress-canary/ingress-canary-x4wh2" Nov 03 21:29:33.700831 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.700810 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkjh7\" (UniqueName: \"kubernetes.io/projected/ddbb8fca-c02a-4e5f-a31f-919194f46f12-kube-api-access-fkjh7\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:33.702603 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.702498 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmt5p\" (UniqueName: \"kubernetes.io/projected/ab012cc6-ec86-41af-b6a1-e9278469c6de-kube-api-access-mmt5p\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hplgb\" (UID: \"ab012cc6-ec86-41af-b6a1-e9278469c6de\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:33.702603 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.702504 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-bound-sa-token\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.702603 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.702589 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz5g7\" (UniqueName: \"kubernetes.io/projected/d8a307b9-8e43-4b73-941d-d5e2c035da14-kube-api-access-dz5g7\") pod \"kas-connection-checker-685895c7c9-7f6xj\" (UID: \"d8a307b9-8e43-4b73-941d-d5e2c035da14\") " pod="kube-system/kas-connection-checker-685895c7c9-7f6xj" Nov 03 21:29:33.702819 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.702622 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pwnr\" (UniqueName: \"kubernetes.io/projected/57dae9a4-2abc-4829-9beb-f61383f46bfe-kube-api-access-8pwnr\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:33.702819 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.702744 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbkft\" (UniqueName: \"kubernetes.io/projected/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-kube-api-access-jbkft\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:33.704753 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.704732 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sldfp\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-kube-api-access-sldfp\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:33.705033 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.705010 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mddx\" (UniqueName: \"kubernetes.io/projected/37a8184c-e9f5-4d96-a816-e0191c32ed06-kube-api-access-6mddx\") pod \"service-ca-operator-544d749c8c-r9qwk\" (UID: \"37a8184c-e9f5-4d96-a816-e0191c32ed06\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:33.705428 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.705407 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nc6h\" (UniqueName: \"kubernetes.io/projected/a734caac-db4c-4f6e-8dd0-415843ecd1b8-kube-api-access-7nc6h\") pod \"kas-connection-checker-685895c7c9-mq2cr\" (UID: \"a734caac-db4c-4f6e-8dd0-415843ecd1b8\") " pod="kube-system/kas-connection-checker-685895c7c9-mq2cr" Nov 03 21:29:33.705698 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.705679 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh5pc\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-kube-api-access-mh5pc\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:33.705967 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.705948 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmdxx\" (UniqueName: \"kubernetes.io/projected/37984c5f-7e58-4323-81f0-4bc7d485ab88-kube-api-access-bmdxx\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.706559 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.706540 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct582\" (UniqueName: \"kubernetes.io/projected/9f173fd7-1ddb-4616-91b2-35a2bd6e0341-kube-api-access-ct582\") pod \"network-check-source-6f78d59f99-m2hhx\" (UID: \"9f173fd7-1ddb-4616-91b2-35a2bd6e0341\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m2hhx" Nov 03 21:29:33.706729 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.706656 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ssv\" (UniqueName: \"kubernetes.io/projected/cb8cd0fc-653b-4f07-a501-f121ec73b7d5-kube-api-access-k8ssv\") pod \"volume-data-source-validator-58c9f5f46d-zthx8\" (UID: \"cb8cd0fc-653b-4f07-a501-f121ec73b7d5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zthx8" Nov 03 21:29:33.706932 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.706914 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btpnr\" (UniqueName: \"kubernetes.io/projected/efc25e42-1fb2-4fb4-b991-ce83b2d174d1-kube-api-access-btpnr\") pod \"kas-connection-checker-685895c7c9-pp685\" (UID: \"efc25e42-1fb2-4fb4-b991-ce83b2d174d1\") " pod="kube-system/kas-connection-checker-685895c7c9-pp685" Nov 03 21:29:33.707250 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.707236 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfcx5\" (UniqueName: \"kubernetes.io/projected/0852ff0f-d078-452f-bb56-3c13512f57ce-kube-api-access-xfcx5\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:33.740309 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.740287 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pp685" Nov 03 21:29:33.750326 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.750308 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7f6xj" Nov 03 21:29:33.763790 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.763773 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mq2cr" Nov 03 21:29:33.802046 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.802014 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:33.806569 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.806524 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zthx8" Nov 03 21:29:33.812889 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.812799 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:33.819810 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.819788 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:33.839054 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.838838 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:33.876031 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.876007 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m2hhx" Nov 03 21:29:33.876276 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:33.876258 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x4wh2" Nov 03 21:29:34.080177 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.080151 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7f6xj"] Nov 03 21:29:34.083216 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.083128 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pp685"] Nov 03 21:29:34.084013 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.083994 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mq2cr"] Nov 03 21:29:34.085637 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:34.085526 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a307b9_8e43_4b73_941d_d5e2c035da14.slice/crio-cfc5b75bf4618eed3366761ac8dbc0b9d0201d7c47c1594e7accd9d259032a89 WatchSource:0}: Error finding container cfc5b75bf4618eed3366761ac8dbc0b9d0201d7c47c1594e7accd9d259032a89: Status 404 returned error can't find the container with id cfc5b75bf4618eed3366761ac8dbc0b9d0201d7c47c1594e7accd9d259032a89 Nov 03 21:29:34.087854 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:34.087831 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda734caac_db4c_4f6e_8dd0_415843ecd1b8.slice/crio-9e0e9f2295547f652724c09ece12e7743ce8ab9e9b7cfd99ceb5365cbe9b4d39 WatchSource:0}: Error finding container 9e0e9f2295547f652724c09ece12e7743ce8ab9e9b7cfd99ceb5365cbe9b4d39: Status 404 returned error can't find the container with id 9e0e9f2295547f652724c09ece12e7743ce8ab9e9b7cfd99ceb5365cbe9b4d39 Nov 03 21:29:34.088450 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:34.088435 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc25e42_1fb2_4fb4_b991_ce83b2d174d1.slice/crio-82d0d86428c0964b3f5d515c5687f67edddfd11e6fff4661560a7c4f27442124 WatchSource:0}: Error finding container 82d0d86428c0964b3f5d515c5687f67edddfd11e6fff4661560a7c4f27442124: Status 404 returned error can't find the container with id 82d0d86428c0964b3f5d515c5687f67edddfd11e6fff4661560a7c4f27442124 Nov 03 21:29:34.167570 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.167464 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wxk7n"] Nov 03 21:29:34.168727 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.168631 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb"] Nov 03 21:29:34.181149 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:34.181125 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab012cc6_ec86_41af_b6a1_e9278469c6de.slice/crio-bd81ef1b72bc14e09804f0dd43588b3bcfdf4bb3fb36206e64c964e68eee2db6 WatchSource:0}: Error finding container bd81ef1b72bc14e09804f0dd43588b3bcfdf4bb3fb36206e64c964e68eee2db6: Status 404 returned error can't find the container with id bd81ef1b72bc14e09804f0dd43588b3bcfdf4bb3fb36206e64c964e68eee2db6 Nov 03 21:29:34.187622 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.187602 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:34.187784 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.187629 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:34.187784 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.187653 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:34.187784 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.187721 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:34.187784 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.187741 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:34.187784 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.187759 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd7fc569c-h759l: secret "image-registry-tls" not found Nov 03 21:29:34.187784 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.187762 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:34.187784 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.187764 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 21:29:34.188026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.187820 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:34.188026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.187837 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf564b6d-c6gsv: secret "image-registry-tls" not found Nov 03 21:29:34.188026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.187837 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 21:29:34.188026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.187807 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls podName:cd1064c9-bade-4061-9b89-5d15e6a26a3d nodeName:}" failed. No retries permitted until 2026-11-03 21:29:35.187789738 +0000 UTC m=+32.079897212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls") pod "image-registry-7fd7fc569c-h759l" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d") : secret "image-registry-tls" not found Nov 03 21:29:34.188026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.187879 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls podName:2a8bade0-e7b8-4d61-b71f-fdb4b2517320 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:35.187867525 +0000 UTC m=+32.079974983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls") pod "dns-default-7t4b8" (UID: "2a8bade0-e7b8-4d61-b71f-fdb4b2517320") : secret "dns-default-metrics-tls" not found Nov 03 21:29:34.188026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.187897 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle podName:0852ff0f-d078-452f-bb56-3c13512f57ce nodeName:}" failed. No retries permitted until 2026-11-03 21:29:35.18788838 +0000 UTC m=+32.079995840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle") pod "router-default-9f5c5b5d5-hpzcq" (UID: "0852ff0f-d078-452f-bb56-3c13512f57ce") : configmap references non-existent config key: service-ca.crt Nov 03 21:29:34.188026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.187922 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls podName:a6467adf-c26e-46ae-a212-98c9846bfe0e nodeName:}" failed. No retries permitted until 2026-11-03 21:29:35.187908137 +0000 UTC m=+32.080015603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls") pod "image-registry-7bf564b6d-c6gsv" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e") : secret "image-registry-tls" not found Nov 03 21:29:34.188026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.187938 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls podName:57dae9a4-2abc-4829-9beb-f61383f46bfe nodeName:}" failed. No retries permitted until 2026-11-03 21:29:35.187932368 +0000 UTC m=+32.080039825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6fc7f" (UID: "57dae9a4-2abc-4829-9beb-f61383f46bfe") : secret "samples-operator-tls" not found Nov 03 21:29:34.188026 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.187965 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:34.188026 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.187989 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:34.188026 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.188017 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:34.188520 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.188061 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:34.188520 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.188086 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 21:29:34.188520 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.188091 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls podName:ddbb8fca-c02a-4e5f-a31f-919194f46f12 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:35.18808218 +0000 UTC m=+32.080189634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wq5rr" (UID: "ddbb8fca-c02a-4e5f-a31f-919194f46f12") : secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:34.188520 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.188100 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 21:29:34.188520 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.188112 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs podName:0852ff0f-d078-452f-bb56-3c13512f57ce nodeName:}" failed. No retries permitted until 2026-11-03 21:29:35.188103672 +0000 UTC m=+32.080211129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs") pod "router-default-9f5c5b5d5-hpzcq" (UID: "0852ff0f-d078-452f-bb56-3c13512f57ce") : secret "router-metrics-certs-default" not found Nov 03 21:29:34.188520 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:34.188158 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert podName:9205351c-9ed0-4a10-8c52-0124c39a20fe nodeName:}" failed. No retries permitted until 2026-11-03 21:29:35.188145003 +0000 UTC m=+32.080252460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w682j" (UID: "9205351c-9ed0-4a10-8c52-0124c39a20fe") : secret "networking-console-plugin-cert" not found Nov 03 21:29:34.211387 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.211332 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-m2hhx"] Nov 03 21:29:34.213158 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.213075 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zthx8"] Nov 03 21:29:34.214564 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.214546 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ltzz4"] Nov 03 21:29:34.215504 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:34.215485 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8cd0fc_653b_4f07_a501_f121ec73b7d5.slice/crio-4869d9aea39cbfef1657dad47963eaed9304d5b8ff2b702f29dd9608d2591b5c WatchSource:0}: Error finding container 4869d9aea39cbfef1657dad47963eaed9304d5b8ff2b702f29dd9608d2591b5c: Status 404 returned error can't find the container with id 4869d9aea39cbfef1657dad47963eaed9304d5b8ff2b702f29dd9608d2591b5c Nov 03 21:29:34.216568 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:34.216553 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f173fd7_1ddb_4616_91b2_35a2bd6e0341.slice/crio-d192bcfde6661cded8df6336b694e6d08725509d75b24e812a37ed3101ca6ceb WatchSource:0}: Error finding container d192bcfde6661cded8df6336b694e6d08725509d75b24e812a37ed3101ca6ceb: Status 404 returned error can't find the container with id d192bcfde6661cded8df6336b694e6d08725509d75b24e812a37ed3101ca6ceb Nov 03 21:29:34.222156 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.222134 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk"] Nov 03 21:29:34.280611 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.280589 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x4wh2"] Nov 03 21:29:34.285045 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:34.285015 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4f0a4f_af3b_4bdc_89f1_b8716c76ded2.slice/crio-c051676d7f856b2e3930b0a6ef88bb2c7007d0086ab1088956499601724dd533 WatchSource:0}: Error finding container c051676d7f856b2e3930b0a6ef88bb2c7007d0086ab1088956499601724dd533: Status 404 returned error can't find the container with id c051676d7f856b2e3930b0a6ef88bb2c7007d0086ab1088956499601724dd533 Nov 03 21:29:34.598653 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.598619 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:34.598653 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.598677 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:34.612938 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.612916 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 21:29:34.613048 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.612984 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bxcrl\"" Nov 03 21:29:34.613048 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.613018 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2kjq7\"" Nov 03 21:29:34.850436 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.850321 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zthx8" event={"ID":"cb8cd0fc-653b-4f07-a501-f121ec73b7d5","Type":"ContainerStarted","Data":"4869d9aea39cbfef1657dad47963eaed9304d5b8ff2b702f29dd9608d2591b5c"} Nov 03 21:29:34.852469 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.852422 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" event={"ID":"ab012cc6-ec86-41af-b6a1-e9278469c6de","Type":"ContainerStarted","Data":"bd81ef1b72bc14e09804f0dd43588b3bcfdf4bb3fb36206e64c964e68eee2db6"} Nov 03 21:29:34.853890 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.853808 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x4wh2" event={"ID":"2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2","Type":"ContainerStarted","Data":"c051676d7f856b2e3930b0a6ef88bb2c7007d0086ab1088956499601724dd533"} Nov 03 21:29:34.856361 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.855955 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pp685" event={"ID":"efc25e42-1fb2-4fb4-b991-ce83b2d174d1","Type":"ContainerStarted","Data":"b239c6bff46989bce85b7756215f5263f4eb016f03829ed311a6200d81010b98"} Nov 03 21:29:34.856361 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.856003 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pp685" event={"ID":"efc25e42-1fb2-4fb4-b991-ce83b2d174d1","Type":"ContainerStarted","Data":"82d0d86428c0964b3f5d515c5687f67edddfd11e6fff4661560a7c4f27442124"} Nov 03 21:29:34.857719 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.857691 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m2hhx" event={"ID":"9f173fd7-1ddb-4616-91b2-35a2bd6e0341","Type":"ContainerStarted","Data":"d192bcfde6661cded8df6336b694e6d08725509d75b24e812a37ed3101ca6ceb"} Nov 03 21:29:34.861208 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.861181 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7f6xj" event={"ID":"d8a307b9-8e43-4b73-941d-d5e2c035da14","Type":"ContainerStarted","Data":"cc1d4537001cc702edd2601302fa1d6bc796400d11a53e7457aec3dfaafe6a21"} Nov 03 21:29:34.861287 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.861219 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7f6xj" event={"ID":"d8a307b9-8e43-4b73-941d-d5e2c035da14","Type":"ContainerStarted","Data":"cfc5b75bf4618eed3366761ac8dbc0b9d0201d7c47c1594e7accd9d259032a89"} Nov 03 21:29:34.863956 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.863933 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" event={"ID":"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b","Type":"ContainerStarted","Data":"27382050184cb35e75ffbf18d7d41ecc433dfa43ae7e28e0f5b45306a34fda3f"} Nov 03 21:29:34.865477 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.865454 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" event={"ID":"37a8184c-e9f5-4d96-a816-e0191c32ed06","Type":"ContainerStarted","Data":"a4d4e4ce5a8adb20888802a97ab5d6b6405195a50b99310b913350660a04735a"} Nov 03 21:29:34.867406 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.867331 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wxk7n" event={"ID":"37984c5f-7e58-4323-81f0-4bc7d485ab88","Type":"ContainerStarted","Data":"25b89988e6a8429f942020dc80ac09dcb82a3d8ccd1f067151e347a4cbd8e040"} Nov 03 21:29:34.869185 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.869161 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mq2cr" event={"ID":"a734caac-db4c-4f6e-8dd0-415843ecd1b8","Type":"ContainerStarted","Data":"74a525bfcfad0d359c47dd43b7f241a67b9fae412e5563b04a8c9fb55afa8d49"} Nov 03 21:29:34.869263 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.869200 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mq2cr" event={"ID":"a734caac-db4c-4f6e-8dd0-415843ecd1b8","Type":"ContainerStarted","Data":"9e0e9f2295547f652724c09ece12e7743ce8ab9e9b7cfd99ceb5365cbe9b4d39"} Nov 03 21:29:34.897217 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.896840 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-pp685" podStartSLOduration=442.896823331 podStartE2EDuration="7m22.896823331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:34.895554559 +0000 UTC m=+31.787662055" watchObservedRunningTime="2026-11-03 21:29:34.896823331 +0000 UTC m=+31.788930807" Nov 03 21:29:34.967434 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.967384 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-mq2cr" podStartSLOduration=442.967371998 podStartE2EDuration="7m22.967371998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:34.967034779 +0000 UTC m=+31.859142254" watchObservedRunningTime="2026-11-03 21:29:34.967371998 +0000 UTC m=+31.859479474" Nov 03 21:29:34.968007 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:34.967967 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7f6xj" podStartSLOduration=442.967958352 podStartE2EDuration="7m22.967958352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:34.931820783 +0000 UTC m=+31.823928261" watchObservedRunningTime="2026-11-03 21:29:34.967958352 +0000 UTC m=+31.860065828" Nov 03 21:29:35.200395 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:35.200315 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:35.200395 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:35.200367 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:35.200395 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:35.200390 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:35.200648 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:35.200424 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:35.200648 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.200439 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 21:29:35.200648 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:35.200492 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:35.200648 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.200520 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs podName:0852ff0f-d078-452f-bb56-3c13512f57ce nodeName:}" failed. No retries permitted until 2026-11-03 21:29:37.200499688 +0000 UTC m=+34.092607158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs") pod "router-default-9f5c5b5d5-hpzcq" (UID: "0852ff0f-d078-452f-bb56-3c13512f57ce") : secret "router-metrics-certs-default" not found Nov 03 21:29:35.200648 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.200576 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 21:29:35.200648 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:35.200587 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:35.200648 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.200624 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls podName:57dae9a4-2abc-4829-9beb-f61383f46bfe nodeName:}" failed. No retries permitted until 2026-11-03 21:29:37.200609753 +0000 UTC m=+34.092717227 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6fc7f" (UID: "57dae9a4-2abc-4829-9beb-f61383f46bfe") : secret "samples-operator-tls" not found Nov 03 21:29:35.201051 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.200685 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle podName:0852ff0f-d078-452f-bb56-3c13512f57ce nodeName:}" failed. No retries permitted until 2026-11-03 21:29:37.200656414 +0000 UTC m=+34.092763903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle") pod "router-default-9f5c5b5d5-hpzcq" (UID: "0852ff0f-d078-452f-bb56-3c13512f57ce") : configmap references non-existent config key: service-ca.crt Nov 03 21:29:35.201051 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:35.200705 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:35.201051 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.200730 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:35.201051 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:35.200737 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:35.201051 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.200742 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd7fc569c-h759l: secret "image-registry-tls" not found Nov 03 21:29:35.201051 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.200794 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 21:29:35.201051 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.200806 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls podName:cd1064c9-bade-4061-9b89-5d15e6a26a3d nodeName:}" failed. No retries permitted until 2026-11-03 21:29:37.200797281 +0000 UTC m=+34.092904756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls") pod "image-registry-7fd7fc569c-h759l" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d") : secret "image-registry-tls" not found Nov 03 21:29:35.201051 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.200829 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls podName:2a8bade0-e7b8-4d61-b71f-fdb4b2517320 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:37.200818163 +0000 UTC m=+34.092925645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls") pod "dns-default-7t4b8" (UID: "2a8bade0-e7b8-4d61-b71f-fdb4b2517320") : secret "dns-default-metrics-tls" not found Nov 03 21:29:35.201051 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.200887 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:35.201051 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.200898 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf564b6d-c6gsv: secret "image-registry-tls" not found Nov 03 21:29:35.201051 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.200932 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls podName:a6467adf-c26e-46ae-a212-98c9846bfe0e nodeName:}" failed. No retries permitted until 2026-11-03 21:29:37.200921818 +0000 UTC m=+34.093029276 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls") pod "image-registry-7bf564b6d-c6gsv" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e") : secret "image-registry-tls" not found Nov 03 21:29:35.201051 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.200983 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 21:29:35.201051 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.201015 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert podName:9205351c-9ed0-4a10-8c52-0124c39a20fe nodeName:}" failed. No retries permitted until 2026-11-03 21:29:37.20100359 +0000 UTC m=+34.093111085 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w682j" (UID: "9205351c-9ed0-4a10-8c52-0124c39a20fe") : secret "networking-console-plugin-cert" not found Nov 03 21:29:35.201617 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.201064 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:35.201617 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:35.201095 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls podName:ddbb8fca-c02a-4e5f-a31f-919194f46f12 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:37.201085436 +0000 UTC m=+34.093192929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wq5rr" (UID: "ddbb8fca-c02a-4e5f-a31f-919194f46f12") : secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:36.885126 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:36.884352 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zthx8" event={"ID":"cb8cd0fc-653b-4f07-a501-f121ec73b7d5","Type":"ContainerStarted","Data":"c21c182943465a0c98f843f58cb7c2fabf3ce7d066acfa611d81c7f83e596040"} Nov 03 21:29:36.922690 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:36.921739 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zthx8" podStartSLOduration=416.829820799 podStartE2EDuration="6m58.921721329s" totalImagesPullingTime="2.09190053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:36.920893024 +0000 UTC m=+33.813000501" watchObservedRunningTime="2026-11-03 21:29:36.921721329 +0000 UTC m=+33.813828839" Nov 03 21:29:37.223026 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:37.222353 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:37.223026 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:37.222403 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:37.223026 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:37.222453 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:37.223026 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:37.222490 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:37.223026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.222566 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 21:29:37.223026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.222581 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:37.223026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.222593 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd7fc569c-h759l: secret "image-registry-tls" not found Nov 03 21:29:37.223026 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:37.222617 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:37.223026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.222644 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls podName:cd1064c9-bade-4061-9b89-5d15e6a26a3d nodeName:}" failed. No retries permitted until 2026-11-03 21:29:41.222628269 +0000 UTC m=+38.114735744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls") pod "image-registry-7fd7fc569c-h759l" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d") : secret "image-registry-tls" not found Nov 03 21:29:37.223026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.222644 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:37.223026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.222711 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls podName:ddbb8fca-c02a-4e5f-a31f-919194f46f12 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:41.22269418 +0000 UTC m=+38.114801655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wq5rr" (UID: "ddbb8fca-c02a-4e5f-a31f-919194f46f12") : secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:37.223026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.222720 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 21:29:37.223026 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:37.222734 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:37.223026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.222768 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls podName:2a8bade0-e7b8-4d61-b71f-fdb4b2517320 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:41.222751946 +0000 UTC m=+38.114859424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls") pod "dns-default-7t4b8" (UID: "2a8bade0-e7b8-4d61-b71f-fdb4b2517320") : secret "dns-default-metrics-tls" not found Nov 03 21:29:37.223026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.222566 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 21:29:37.223026 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.222787 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert podName:9205351c-9ed0-4a10-8c52-0124c39a20fe nodeName:}" failed. No retries permitted until 2026-11-03 21:29:41.222777727 +0000 UTC m=+38.114885189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w682j" (UID: "9205351c-9ed0-4a10-8c52-0124c39a20fe") : secret "networking-console-plugin-cert" not found Nov 03 21:29:37.223966 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.222811 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs podName:0852ff0f-d078-452f-bb56-3c13512f57ce nodeName:}" failed. No retries permitted until 2026-11-03 21:29:41.222803001 +0000 UTC m=+38.114910489 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs") pod "router-default-9f5c5b5d5-hpzcq" (UID: "0852ff0f-d078-452f-bb56-3c13512f57ce") : secret "router-metrics-certs-default" not found Nov 03 21:29:37.223966 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.222900 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:37.223966 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.223007 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf564b6d-c6gsv: secret "image-registry-tls" not found Nov 03 21:29:37.223966 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:37.223043 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:37.223966 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.223097 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls podName:a6467adf-c26e-46ae-a212-98c9846bfe0e nodeName:}" failed. No retries permitted until 2026-11-03 21:29:41.223085856 +0000 UTC m=+38.115193311 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls") pod "image-registry-7bf564b6d-c6gsv" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e") : secret "image-registry-tls" not found Nov 03 21:29:37.223966 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.223122 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 21:29:37.223966 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.223168 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls podName:57dae9a4-2abc-4829-9beb-f61383f46bfe nodeName:}" failed. No retries permitted until 2026-11-03 21:29:41.223155531 +0000 UTC m=+38.115262991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6fc7f" (UID: "57dae9a4-2abc-4829-9beb-f61383f46bfe") : secret "samples-operator-tls" not found Nov 03 21:29:37.223966 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:37.223204 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:37.223966 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.223333 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle podName:0852ff0f-d078-452f-bb56-3c13512f57ce nodeName:}" failed. No retries permitted until 2026-11-03 21:29:41.223322543 +0000 UTC m=+38.115430024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle") pod "router-default-9f5c5b5d5-hpzcq" (UID: "0852ff0f-d078-452f-bb56-3c13512f57ce") : configmap references non-existent config key: service-ca.crt Nov 03 21:29:37.426480 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:37.425884 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs\") pod \"network-metrics-daemon-qvv5p\" (UID: \"2a406a0d-024a-411f-94c2-953a739c214f\") " pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:37.426480 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.426094 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 21:29:37.426480 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:37.426162 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs podName:2a406a0d-024a-411f-94c2-953a739c214f nodeName:}" failed. No retries permitted until 2026-11-03 21:30:09.426142626 +0000 UTC m=+66.318250097 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs") pod "network-metrics-daemon-qvv5p" (UID: "2a406a0d-024a-411f-94c2-953a739c214f") : secret "metrics-daemon-secret" not found Nov 03 21:29:37.528366 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:37.527471 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dl8\" (UniqueName: \"kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8\") pod \"network-check-target-pllqd\" (UID: \"ca003584-cd18-4740-a619-66c89853c773\") " pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:37.528366 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:37.527529 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:37.530790 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:37.530759 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:37.531600 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:37.531575 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7dl8\" (UniqueName: \"kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8\") pod \"network-check-target-pllqd\" (UID: \"ca003584-cd18-4740-a619-66c89853c773\") " pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:37.625852 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:37.625819 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:37.812324 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:37.812283 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:37.964936 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:37.964851 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pllqd"] Nov 03 21:29:38.037904 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:38.037817 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca003584_cd18_4740_a619_66c89853c773.slice/crio-60cd9a89ee846b5d7caeff05ada37137631b71e8a299d752fff10bb925cee194 WatchSource:0}: Error finding container 60cd9a89ee846b5d7caeff05ada37137631b71e8a299d752fff10bb925cee194: Status 404 returned error can't find the container with id 60cd9a89ee846b5d7caeff05ada37137631b71e8a299d752fff10bb925cee194 Nov 03 21:29:38.116187 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:38.116160 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z5jjd"] Nov 03 21:29:38.894035 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:38.893452 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/0.log" Nov 03 21:29:38.894035 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:38.893489 2584 generic.go:356] "Generic (PLEG): container finished" podID="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" containerID="a34e89d61b8c602148f0f97cd891e95e671c6ec7f45d4faefc4ef1abf5e76907" exitCode=255 Nov 03 21:29:38.894035 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:38.893579 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" event={"ID":"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b","Type":"ContainerDied","Data":"a34e89d61b8c602148f0f97cd891e95e671c6ec7f45d4faefc4ef1abf5e76907"} Nov 03 21:29:38.894035 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:38.893880 2584 scope.go:122] "RemoveContainer" containerID="a34e89d61b8c602148f0f97cd891e95e671c6ec7f45d4faefc4ef1abf5e76907" Nov 03 21:29:38.895360 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:38.895335 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" event={"ID":"ab012cc6-ec86-41af-b6a1-e9278469c6de","Type":"ContainerStarted","Data":"565f24b8c71c179303ae03af26b133b4a5bb60d9b29a51996f08d35a2ef4de52"} Nov 03 21:29:38.896872 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:38.896848 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z5jjd" event={"ID":"58ec685b-55f0-42a2-9112-6fe8a813fd8a","Type":"ContainerStarted","Data":"a35ad8d6727d685ad030cfba62fc44d2a35788cee701cb6b430e82037708f095"} Nov 03 21:29:38.900353 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:38.900331 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pllqd" event={"ID":"ca003584-cd18-4740-a619-66c89853c773","Type":"ContainerStarted","Data":"60cd9a89ee846b5d7caeff05ada37137631b71e8a299d752fff10bb925cee194"} Nov 03 21:29:39.837379 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.837326 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tggjh"] Nov 03 21:29:39.860333 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.860301 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-h2s5x"] Nov 03 21:29:39.861751 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.861597 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tggjh" Nov 03 21:29:39.878720 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.878614 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-th7gv\"" Nov 03 21:29:39.878861 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.878818 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 21:29:39.878917 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.878868 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 21:29:39.886833 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.886719 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tggjh"] Nov 03 21:29:39.886833 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.886765 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-h2s5x"] Nov 03 21:29:39.886967 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.886860 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h2s5x" Nov 03 21:29:39.917110 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.917078 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x4wh2" event={"ID":"2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2","Type":"ContainerStarted","Data":"e71c138f93830e375bb3cf54a9e079d4d9af7171465c7789b37d8ca36d865e93"} Nov 03 21:29:39.919313 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.919290 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/0.log" Nov 03 21:29:39.919434 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.919362 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" event={"ID":"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b","Type":"ContainerStarted","Data":"5b4e56a0a125fb45430d89a491d2d78b22f384f0c1b89cd5dec41837a5111987"} Nov 03 21:29:39.920096 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.920078 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:39.925524 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.924901 2584 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-ltzz4 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body= Nov 03 21:29:39.925524 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.924949 2584 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" podUID="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" Nov 03 21:29:39.926656 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.926630 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" event={"ID":"37a8184c-e9f5-4d96-a816-e0191c32ed06","Type":"ContainerStarted","Data":"e0b15d83f86ee8174c763d961444733403907502b2abde73dc05af24a76e6bc5"} Nov 03 21:29:39.930691 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.930614 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wxk7n" event={"ID":"37984c5f-7e58-4323-81f0-4bc7d485ab88","Type":"ContainerStarted","Data":"8cb75a53ae18d6a72660e08550ff834f5d5d0a0d7912eeb0ac4c8a859271eef9"} Nov 03 21:29:39.951987 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:39.951563 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phnxv\" (UniqueName: \"kubernetes.io/projected/9a35bad8-8af4-40c5-aa2b-1a939cc72c5e-kube-api-access-phnxv\") pod \"migrator-66df5d496b-tggjh\" (UID: \"9a35bad8-8af4-40c5-aa2b-1a939cc72c5e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tggjh" Nov 03 21:29:40.018055 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.018003 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" podStartSLOduration=417.410628527 podStartE2EDuration="7m2.017984223s" totalImagesPullingTime="4.607355696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:39.97547054 +0000 UTC m=+36.867578020" watchObservedRunningTime="2026-11-03 21:29:40.017984223 +0000 UTC m=+36.910091697" Nov 03 21:29:40.018649 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.018441 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x4wh2" podStartSLOduration=1.733844921 podStartE2EDuration="7.018429928s" totalImagesPullingTime="5.284585007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:40.015629943 +0000 UTC m=+36.907737418" watchObservedRunningTime="2026-11-03 21:29:40.018429928 +0000 UTC m=+36.910537399" Nov 03 21:29:40.052883 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.052856 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phnxv\" (UniqueName: \"kubernetes.io/projected/9a35bad8-8af4-40c5-aa2b-1a939cc72c5e-kube-api-access-phnxv\") pod \"migrator-66df5d496b-tggjh\" (UID: \"9a35bad8-8af4-40c5-aa2b-1a939cc72c5e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tggjh" Nov 03 21:29:40.052999 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.052972 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8srsk\" (UniqueName: \"kubernetes.io/projected/43fb3f37-13b3-4eaa-9b92-6c343b1b2cef-kube-api-access-8srsk\") pod \"migrator-66df5d496b-h2s5x\" (UID: \"43fb3f37-13b3-4eaa-9b92-6c343b1b2cef\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h2s5x" Nov 03 21:29:40.054359 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.054310 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-wxk7n" podStartSLOduration=437.68913697200003 podStartE2EDuration="7m23.054294451s" totalImagesPullingTime="5.365157479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:40.052415133 +0000 UTC m=+36.944522610" watchObservedRunningTime="2026-11-03 21:29:40.054294451 +0000 UTC m=+36.946401927" Nov 03 21:29:40.076208 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.076142 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phnxv\" (UniqueName: \"kubernetes.io/projected/9a35bad8-8af4-40c5-aa2b-1a939cc72c5e-kube-api-access-phnxv\") pod \"migrator-66df5d496b-tggjh\" (UID: \"9a35bad8-8af4-40c5-aa2b-1a939cc72c5e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tggjh" Nov 03 21:29:40.090034 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.089984 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" podStartSLOduration=415.78995154 podStartE2EDuration="7m1.089967635s" totalImagesPullingTime="5.300016095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:40.088725921 +0000 UTC m=+36.980833399" watchObservedRunningTime="2026-11-03 21:29:40.089967635 +0000 UTC m=+36.982075110" Nov 03 21:29:40.128628 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.127639 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" podStartSLOduration=417.313555703 podStartE2EDuration="7m1.127627933s" totalImagesPullingTime="3.81407223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:40.12714078 +0000 UTC m=+37.019248259" watchObservedRunningTime="2026-11-03 21:29:40.127627933 +0000 UTC m=+37.019735409" Nov 03 21:29:40.154372 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.154342 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srsk\" (UniqueName: \"kubernetes.io/projected/43fb3f37-13b3-4eaa-9b92-6c343b1b2cef-kube-api-access-8srsk\") pod \"migrator-66df5d496b-h2s5x\" (UID: \"43fb3f37-13b3-4eaa-9b92-6c343b1b2cef\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h2s5x" Nov 03 21:29:40.176768 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.176739 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srsk\" (UniqueName: \"kubernetes.io/projected/43fb3f37-13b3-4eaa-9b92-6c343b1b2cef-kube-api-access-8srsk\") pod \"migrator-66df5d496b-h2s5x\" (UID: \"43fb3f37-13b3-4eaa-9b92-6c343b1b2cef\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h2s5x" Nov 03 21:29:40.196875 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.196657 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tggjh" Nov 03 21:29:40.216929 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.216906 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h2s5x" Nov 03 21:29:40.542210 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.542176 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tggjh"] Nov 03 21:29:40.549969 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:40.549924 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a35bad8_8af4_40c5_aa2b_1a939cc72c5e.slice/crio-07c53b8d563f478b77c7b6d440a45adb2b6e6cb433b25dd1d2ec2fad25ce7e2b WatchSource:0}: Error finding container 07c53b8d563f478b77c7b6d440a45adb2b6e6cb433b25dd1d2ec2fad25ce7e2b: Status 404 returned error can't find the container with id 07c53b8d563f478b77c7b6d440a45adb2b6e6cb433b25dd1d2ec2fad25ce7e2b Nov 03 21:29:40.555354 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.555327 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-h2s5x"] Nov 03 21:29:40.559631 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:40.559606 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43fb3f37_13b3_4eaa_9b92_6c343b1b2cef.slice/crio-2ad3d8d6cdc0c805825cf44a5a711406ad557fbb9b6a79930ff3f0e0d742178f WatchSource:0}: Error finding container 2ad3d8d6cdc0c805825cf44a5a711406ad557fbb9b6a79930ff3f0e0d742178f: Status 404 returned error can't find the container with id 2ad3d8d6cdc0c805825cf44a5a711406ad557fbb9b6a79930ff3f0e0d742178f Nov 03 21:29:40.584306 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.584254 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-plrfk"] Nov 03 21:29:40.608038 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.608015 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-plrfk"] Nov 03 21:29:40.608176 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.608138 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:40.625266 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.625240 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 21:29:40.625383 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.625242 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 21:29:40.625552 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.625539 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qcvh8\"" Nov 03 21:29:40.653311 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.653286 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-dflhb-2sf79"] Nov 03 21:29:40.668841 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.668817 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:40.675413 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.674484 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dflhb-2sf79"] Nov 03 21:29:40.684315 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.684297 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lj4ds\"" Nov 03 21:29:40.760728 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.760691 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-crio-socket\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:40.760919 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.760754 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-data-volume\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:40.760919 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.760789 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ktf5\" (UniqueName: \"kubernetes.io/projected/ba86e754-0fcc-46ab-a5e2-10609342eb13-kube-api-access-5ktf5\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:40.760919 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.760822 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ba86e754-0fcc-46ab-a5e2-10609342eb13-archives-path\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:40.760919 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.760848 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-trusted-ca-bundle\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:40.760919 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.760898 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:40.761170 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.760932 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:40.761170 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.760970 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-service-ca-bundle\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:40.761170 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.761033 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2jlb\" (UniqueName: \"kubernetes.io/projected/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-api-access-n2jlb\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:40.862082 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.862001 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-crio-socket\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:40.862082 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.862062 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-data-volume\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:40.862524 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.862092 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ktf5\" (UniqueName: \"kubernetes.io/projected/ba86e754-0fcc-46ab-a5e2-10609342eb13-kube-api-access-5ktf5\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:40.862524 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.862122 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ba86e754-0fcc-46ab-a5e2-10609342eb13-archives-path\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:40.862524 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.862139 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-trusted-ca-bundle\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:40.862524 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.862226 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:40.862524 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.862262 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:40.862524 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.862289 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-service-ca-bundle\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:40.862524 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.862335 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2jlb\" (UniqueName: \"kubernetes.io/projected/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-api-access-n2jlb\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:40.862812 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.862650 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-data-volume\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:40.862812 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.862146 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-crio-socket\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:40.862812 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:40.862784 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 21:29:40.862812 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:40.862795 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 21:29:40.862995 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:40.862843 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-rbac-proxy-cm podName:7a8ffb5d-6f44-499a-b1bc-1cf127993dc7 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:41.36282579 +0000 UTC m=+38.254933251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-plrfk" (UID: "7a8ffb5d-6f44-499a-b1bc-1cf127993dc7") : configmap "kube-rbac-proxy" not found Nov 03 21:29:40.862995 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:40.862857 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls podName:7a8ffb5d-6f44-499a-b1bc-1cf127993dc7 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:41.36284982 +0000 UTC m=+38.254957284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-plrfk" (UID: "7a8ffb5d-6f44-499a-b1bc-1cf127993dc7") : secret "insights-runtime-extractor-tls" not found Nov 03 21:29:40.862995 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.862905 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ba86e754-0fcc-46ab-a5e2-10609342eb13-archives-path\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:40.863332 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.863287 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-service-ca-bundle\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:40.863482 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.863461 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-trusted-ca-bundle\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:40.883939 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.883892 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ktf5\" (UniqueName: \"kubernetes.io/projected/ba86e754-0fcc-46ab-a5e2-10609342eb13-kube-api-access-5ktf5\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:40.893754 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.893728 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2jlb\" (UniqueName: \"kubernetes.io/projected/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-api-access-n2jlb\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:40.934858 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.934822 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h2s5x" event={"ID":"43fb3f37-13b3-4eaa-9b92-6c343b1b2cef","Type":"ContainerStarted","Data":"2ad3d8d6cdc0c805825cf44a5a711406ad557fbb9b6a79930ff3f0e0d742178f"} Nov 03 21:29:40.936432 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.936403 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m2hhx" event={"ID":"9f173fd7-1ddb-4616-91b2-35a2bd6e0341","Type":"ContainerStarted","Data":"2de28fc29727e9031e8fe89980fd494cbcb7e9dd2bfbb1c40f79f4132284ec98"} Nov 03 21:29:40.937991 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.937936 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tggjh" event={"ID":"9a35bad8-8af4-40c5-aa2b-1a939cc72c5e","Type":"ContainerStarted","Data":"07c53b8d563f478b77c7b6d440a45adb2b6e6cb433b25dd1d2ec2fad25ce7e2b"} Nov 03 21:29:40.938400 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.938377 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-rg8rf"] Nov 03 21:29:40.939906 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.939883 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/1.log" Nov 03 21:29:40.940336 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.940319 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/0.log" Nov 03 21:29:40.940423 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.940350 2584 generic.go:356] "Generic (PLEG): container finished" podID="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" containerID="5b4e56a0a125fb45430d89a491d2d78b22f384f0c1b89cd5dec41837a5111987" exitCode=255 Nov 03 21:29:40.941074 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.941052 2584 scope.go:122] "RemoveContainer" containerID="5b4e56a0a125fb45430d89a491d2d78b22f384f0c1b89cd5dec41837a5111987" Nov 03 21:29:40.941252 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:40.941231 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ltzz4_openshift-console-operator(c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" podUID="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" Nov 03 21:29:40.952528 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.952507 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" event={"ID":"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b","Type":"ContainerDied","Data":"5b4e56a0a125fb45430d89a491d2d78b22f384f0c1b89cd5dec41837a5111987"} Nov 03 21:29:40.952621 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.952540 2584 scope.go:122] "RemoveContainer" containerID="a34e89d61b8c602148f0f97cd891e95e671c6ec7f45d4faefc4ef1abf5e76907" Nov 03 21:29:40.952704 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.952629 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:40.963278 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.963259 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-rg8rf"] Nov 03 21:29:40.969186 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.968406 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 21:29:40.973749 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.973716 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nz5lt\"" Nov 03 21:29:40.973749 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.973742 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 21:29:40.973880 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.973790 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 21:29:40.974671 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:40.974611 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m2hhx" podStartSLOduration=415.209925445 podStartE2EDuration="7m0.974593317s" totalImagesPullingTime="5.764667872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:40.974034592 +0000 UTC m=+37.866142069" watchObservedRunningTime="2026-11-03 21:29:40.974593317 +0000 UTC m=+37.866700795" Nov 03 21:29:41.003191 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.003170 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:41.067796 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.065031 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3d339ae6-a9e4-4258-8fe1-a393b943f953-signing-key\") pod \"service-ca-5fffc97bdf-rg8rf\" (UID: \"3d339ae6-a9e4-4258-8fe1-a393b943f953\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:41.067796 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.065150 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8fx4\" (UniqueName: \"kubernetes.io/projected/3d339ae6-a9e4-4258-8fe1-a393b943f953-kube-api-access-k8fx4\") pod \"service-ca-5fffc97bdf-rg8rf\" (UID: \"3d339ae6-a9e4-4258-8fe1-a393b943f953\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:41.067796 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.065312 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3d339ae6-a9e4-4258-8fe1-a393b943f953-signing-cabundle\") pod \"service-ca-5fffc97bdf-rg8rf\" (UID: \"3d339ae6-a9e4-4258-8fe1-a393b943f953\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:41.068420 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.068401 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 21:29:41.166849 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.166779 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8fx4\" (UniqueName: \"kubernetes.io/projected/3d339ae6-a9e4-4258-8fe1-a393b943f953-kube-api-access-k8fx4\") pod \"service-ca-5fffc97bdf-rg8rf\" (UID: \"3d339ae6-a9e4-4258-8fe1-a393b943f953\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:41.166990 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.166868 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3d339ae6-a9e4-4258-8fe1-a393b943f953-signing-cabundle\") pod \"service-ca-5fffc97bdf-rg8rf\" (UID: \"3d339ae6-a9e4-4258-8fe1-a393b943f953\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:41.166990 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.166975 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3d339ae6-a9e4-4258-8fe1-a393b943f953-signing-key\") pod \"service-ca-5fffc97bdf-rg8rf\" (UID: \"3d339ae6-a9e4-4258-8fe1-a393b943f953\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:41.167525 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.167500 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3d339ae6-a9e4-4258-8fe1-a393b943f953-signing-cabundle\") pod \"service-ca-5fffc97bdf-rg8rf\" (UID: \"3d339ae6-a9e4-4258-8fe1-a393b943f953\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:41.169424 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.169395 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3d339ae6-a9e4-4258-8fe1-a393b943f953-signing-key\") pod \"service-ca-5fffc97bdf-rg8rf\" (UID: \"3d339ae6-a9e4-4258-8fe1-a393b943f953\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:41.185748 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.185708 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8fx4\" (UniqueName: \"kubernetes.io/projected/3d339ae6-a9e4-4258-8fe1-a393b943f953-kube-api-access-k8fx4\") pod \"service-ca-5fffc97bdf-rg8rf\" (UID: \"3d339ae6-a9e4-4258-8fe1-a393b943f953\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:41.268290 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.268255 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:41.268401 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.268304 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:41.268401 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.268342 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:41.268494 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.268397 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:41.268494 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.268469 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:41.268559 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.268548 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:41.268921 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.268595 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:41.268921 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.268646 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:41.268921 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268714 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:41.268921 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268724 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:41.268921 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268747 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd7fc569c-h759l: secret "image-registry-tls" not found Nov 03 21:29:41.268921 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268733 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle podName:0852ff0f-d078-452f-bb56-3c13512f57ce nodeName:}" failed. No retries permitted until 2026-11-03 21:29:49.268712498 +0000 UTC m=+46.160819966 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle") pod "router-default-9f5c5b5d5-hpzcq" (UID: "0852ff0f-d078-452f-bb56-3c13512f57ce") : configmap references non-existent config key: service-ca.crt Nov 03 21:29:41.268921 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268789 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 21:29:41.268921 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268806 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls podName:cd1064c9-bade-4061-9b89-5d15e6a26a3d nodeName:}" failed. No retries permitted until 2026-11-03 21:29:49.268794178 +0000 UTC m=+46.160901633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls") pod "image-registry-7fd7fc569c-h759l" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d") : secret "image-registry-tls" not found Nov 03 21:29:41.268921 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268837 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls podName:2a8bade0-e7b8-4d61-b71f-fdb4b2517320 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:49.268825239 +0000 UTC m=+46.160932714 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls") pod "dns-default-7t4b8" (UID: "2a8bade0-e7b8-4d61-b71f-fdb4b2517320") : secret "dns-default-metrics-tls" not found Nov 03 21:29:41.268921 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268840 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:41.268921 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268847 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 21:29:41.268921 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268872 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert podName:9205351c-9ed0-4a10-8c52-0124c39a20fe nodeName:}" failed. No retries permitted until 2026-11-03 21:29:49.268865594 +0000 UTC m=+46.160973048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w682j" (UID: "9205351c-9ed0-4a10-8c52-0124c39a20fe") : secret "networking-console-plugin-cert" not found Nov 03 21:29:41.268921 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268735 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf564b6d-c6gsv: secret "image-registry-tls" not found Nov 03 21:29:41.268921 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268891 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 21:29:41.268921 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268790 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 21:29:41.268921 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268883 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls podName:ddbb8fca-c02a-4e5f-a31f-919194f46f12 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:49.268877907 +0000 UTC m=+46.160985361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wq5rr" (UID: "ddbb8fca-c02a-4e5f-a31f-919194f46f12") : secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:41.269761 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268943 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls podName:a6467adf-c26e-46ae-a212-98c9846bfe0e nodeName:}" failed. No retries permitted until 2026-11-03 21:29:49.268931584 +0000 UTC m=+46.161039045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls") pod "image-registry-7bf564b6d-c6gsv" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e") : secret "image-registry-tls" not found Nov 03 21:29:41.269761 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268966 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls podName:57dae9a4-2abc-4829-9beb-f61383f46bfe nodeName:}" failed. No retries permitted until 2026-11-03 21:29:49.26895808 +0000 UTC m=+46.161065550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6fc7f" (UID: "57dae9a4-2abc-4829-9beb-f61383f46bfe") : secret "samples-operator-tls" not found Nov 03 21:29:41.269761 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.268978 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs podName:0852ff0f-d078-452f-bb56-3c13512f57ce nodeName:}" failed. No retries permitted until 2026-11-03 21:29:49.268971848 +0000 UTC m=+46.161079303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs") pod "router-default-9f5c5b5d5-hpzcq" (UID: "0852ff0f-d078-452f-bb56-3c13512f57ce") : secret "router-metrics-certs-default" not found Nov 03 21:29:41.274448 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.274429 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:41.329691 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.327648 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dflhb-2sf79"] Nov 03 21:29:41.370702 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.369235 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:41.370702 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.369284 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:41.370702 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.369469 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 21:29:41.370702 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.369526 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-rbac-proxy-cm podName:7a8ffb5d-6f44-499a-b1bc-1cf127993dc7 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:42.369509259 +0000 UTC m=+39.261616726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-plrfk" (UID: "7a8ffb5d-6f44-499a-b1bc-1cf127993dc7") : configmap "kube-rbac-proxy" not found Nov 03 21:29:41.370702 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.369607 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 21:29:41.370702 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.369642 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls podName:7a8ffb5d-6f44-499a-b1bc-1cf127993dc7 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:42.369631734 +0000 UTC m=+39.261739190 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-plrfk" (UID: "7a8ffb5d-6f44-499a-b1bc-1cf127993dc7") : secret "insights-runtime-extractor-tls" not found Nov 03 21:29:41.613912 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.613881 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-rg8rf"] Nov 03 21:29:41.619575 ip-10-0-131-253 kubenswrapper[2584]: W1103 21:29:41.619542 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d339ae6_a9e4_4258_8fe1_a393b943f953.slice/crio-812f5bfd753120fc01dee8828ddc157907904f1ed900462e48a4d9126db9ab99 WatchSource:0}: Error finding container 812f5bfd753120fc01dee8828ddc157907904f1ed900462e48a4d9126db9ab99: Status 404 returned error can't find the container with id 812f5bfd753120fc01dee8828ddc157907904f1ed900462e48a4d9126db9ab99 Nov 03 21:29:41.946685 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.946042 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pllqd" event={"ID":"ca003584-cd18-4740-a619-66c89853c773","Type":"ContainerStarted","Data":"27d719eaceef69cf2da0304b26b9d9b6186f86a5247b58a46e6d3977c8afed7f"} Nov 03 21:29:41.947556 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.947390 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:41.949075 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.948713 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dflhb-2sf79" event={"ID":"ba86e754-0fcc-46ab-a5e2-10609342eb13","Type":"ContainerStarted","Data":"e16de07c8a8d6c890d059e74651fd941023879c54134540c9620a847e99d1dfd"} Nov 03 21:29:41.949075 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.948751 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dflhb-2sf79" event={"ID":"ba86e754-0fcc-46ab-a5e2-10609342eb13","Type":"ContainerStarted","Data":"bcf1d521ab66e7d26743f70d0164ebeb81aef32e0cea06893d82ceb83b8ce414"} Nov 03 21:29:41.950840 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.950784 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/1.log" Nov 03 21:29:41.951157 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.951144 2584 scope.go:122] "RemoveContainer" containerID="5b4e56a0a125fb45430d89a491d2d78b22f384f0c1b89cd5dec41837a5111987" Nov 03 21:29:41.951333 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:41.951310 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ltzz4_openshift-console-operator(c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" podUID="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" Nov 03 21:29:41.953851 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.953760 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" event={"ID":"3d339ae6-a9e4-4258-8fe1-a393b943f953","Type":"ContainerStarted","Data":"9fba28068da494651bdcb060e06f5404c5fb21c37d64cb29d83c67857bc8a19f"} Nov 03 21:29:41.953851 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:41.953799 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" event={"ID":"3d339ae6-a9e4-4258-8fe1-a393b943f953","Type":"ContainerStarted","Data":"812f5bfd753120fc01dee8828ddc157907904f1ed900462e48a4d9126db9ab99"} Nov 03 21:29:42.022632 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:42.022568 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pllqd" podStartSLOduration=34.846987013 podStartE2EDuration="38.022550605s" totalImagesPullingTime="3.175563592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:41.98569522 +0000 UTC m=+38.877802698" watchObservedRunningTime="2026-11-03 21:29:42.022550605 +0000 UTC m=+38.914658083" Nov 03 21:29:42.022899 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:42.022862 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" podStartSLOduration=2.022850266 podStartE2EDuration="2.022850266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:42.021206969 +0000 UTC m=+38.913314441" watchObservedRunningTime="2026-11-03 21:29:42.022850266 +0000 UTC m=+38.914957743" Nov 03 21:29:42.059012 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:42.058290 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-dflhb-2sf79" podStartSLOduration=2.058268543 podStartE2EDuration="2.058268543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:42.056836901 +0000 UTC m=+38.948944377" watchObservedRunningTime="2026-11-03 21:29:42.058268543 +0000 UTC m=+38.950376020" Nov 03 21:29:42.382545 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:42.382503 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:42.382708 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:42.382556 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:42.382708 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:42.382691 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 21:29:42.382777 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:42.382715 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 21:29:42.382777 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:42.382771 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls podName:7a8ffb5d-6f44-499a-b1bc-1cf127993dc7 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:44.382751946 +0000 UTC m=+41.274859406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-plrfk" (UID: "7a8ffb5d-6f44-499a-b1bc-1cf127993dc7") : secret "insights-runtime-extractor-tls" not found Nov 03 21:29:42.382940 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:42.382791 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-rbac-proxy-cm podName:7a8ffb5d-6f44-499a-b1bc-1cf127993dc7 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:44.382781112 +0000 UTC m=+41.274888575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-plrfk" (UID: "7a8ffb5d-6f44-499a-b1bc-1cf127993dc7") : configmap "kube-rbac-proxy" not found Nov 03 21:29:42.967751 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:42.967711 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h2s5x" event={"ID":"43fb3f37-13b3-4eaa-9b92-6c343b1b2cef","Type":"ContainerStarted","Data":"9ffae8c3a7c4f5b9df3a5621268ab5d8d8ef5c8da6cab176ef58db1027e51974"} Nov 03 21:29:42.970040 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:42.970011 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tggjh" event={"ID":"9a35bad8-8af4-40c5-aa2b-1a939cc72c5e","Type":"ContainerStarted","Data":"a21f1adf53fb6a7d2ea2e4680bf42c1ac26fee3ce29dfdfbbbb70b02519c90a8"} Nov 03 21:29:42.971260 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:42.970779 2584 scope.go:122] "RemoveContainer" containerID="5b4e56a0a125fb45430d89a491d2d78b22f384f0c1b89cd5dec41837a5111987" Nov 03 21:29:42.971260 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:42.970916 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ltzz4_openshift-console-operator(c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" podUID="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" Nov 03 21:29:43.010562 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:43.010503 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h2s5x" podStartSLOduration=1.969600626 podStartE2EDuration="4.010485589s" totalImagesPullingTime="2.040884963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:43.008954941 +0000 UTC m=+39.901062418" watchObservedRunningTime="2026-11-03 21:29:43.010485589 +0000 UTC m=+39.902593066" Nov 03 21:29:43.048795 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:43.048298 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tggjh" podStartSLOduration=1.999604615 podStartE2EDuration="4.048278625s" totalImagesPullingTime="2.04867401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:43.047301756 +0000 UTC m=+39.939409230" watchObservedRunningTime="2026-11-03 21:29:43.048278625 +0000 UTC m=+39.940386103" Nov 03 21:29:43.840109 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:43.840077 2584 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:43.973860 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:43.973833 2584 scope.go:122] "RemoveContainer" containerID="5b4e56a0a125fb45430d89a491d2d78b22f384f0c1b89cd5dec41837a5111987" Nov 03 21:29:43.974286 ip-10-0-131-253 kubenswrapper[2584]: E1103 21:29:43.974029 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ltzz4_openshift-console-operator(c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" podUID="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" Nov 03 21:29:44.406721 ip-10-0-131-253 kubenswrapper[2584]: I1103 21:29:44.406638 2584 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 21:29:44.406783 ip-10-0-131-253 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 21:29:44.430023 ip-10-0-131-253 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 21:29:44.430188 ip-10-0-131-253 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 21:29:44.430235 ip-10-0-131-253 systemd[1]: kubelet.service: Consumed 3.060s CPU time. Nov 03 21:29:44.449126 ip-10-0-131-253 systemd[1]: Starting Kubernetes Kubelet... Nov 03 21:29:44.530490 ip-10-0-131-253 kubenswrapper[5041]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:29:44.530490 ip-10-0-131-253 kubenswrapper[5041]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 21:29:44.530490 ip-10-0-131-253 kubenswrapper[5041]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:29:44.530490 ip-10-0-131-253 kubenswrapper[5041]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 21:29:44.533550 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533453 5041 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:29:44.533550 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533547 5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:29:44.533550 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533551 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:29:44.533550 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533554 5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:29:44.533550 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533558 5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533561 5041 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533565 5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533567 5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533570 5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533573 5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533575 5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533578 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533581 5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533583 5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533586 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533588 5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533591 5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533594 5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533596 5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533607 5041 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533610 5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533612 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533616 5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533618 5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:29:44.533845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533621 5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533624 5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533627 5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533629 5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533632 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533634 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533637 5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533639 5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533644 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533647 5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533649 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533652 5041 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533654 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533672 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533677 5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533681 5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533686 5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533689 5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533691 5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533695 5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:29:44.534621 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533697 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533700 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533705 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533710 5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533714 5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533717 5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533722 5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533728 5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533734 5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533738 5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533742 5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533747 5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533753 5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533757 5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533760 5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533763 5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533766 5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533768 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533771 5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:29:44.535419 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533773 5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533784 5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533788 5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533792 5041 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533794 5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533797 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533800 5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533803 5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533805 5041 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533808 5041 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533810 5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533813 5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533815 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533818 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533820 5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533823 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533825 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533828 5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533831 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533834 5041 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:29:44.536171 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533837 5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533839 5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533843 5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533845 5041 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533848 5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533852 5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533855 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533858 5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533861 5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533863 5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533866 5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533869 5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533871 5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533875 5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533878 5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533883 5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533886 5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533890 5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533892 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:29:44.536969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533895 5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533898 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533900 5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.533903 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534674 5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534683 5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534687 5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534689 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534692 5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534695 5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534698 5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534700 5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534703 5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534706 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534709 5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534712 5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534715 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534717 5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534722 5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:29:44.537751 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534725 5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534728 5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534731 5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534733 5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534736 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534738 5041 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534741 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534745 5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534748 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534750 5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534753 5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534756 5041 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534758 5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534761 5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534763 5041 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534766 5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534768 5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534771 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534773 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534776 5041 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:29:44.538502 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534779 5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534781 5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534784 5041 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534786 5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534789 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534791 5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534794 5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534797 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534800 5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534802 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534805 5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534808 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534811 5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534813 5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534816 5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534818 5041 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534821 5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534823 5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534826 5041 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534830 5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:29:44.539306 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534832 5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534835 5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534837 5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534840 5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534842 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534845 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534848 5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534850 5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534853 5041 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534855 5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534858 5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534860 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534864 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534866 5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534869 5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534871 5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534874 5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534876 5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534879 5041 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534881 5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:29:44.540137 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534884 5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534888 5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534892 5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534896 5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534899 5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534902 5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534904 5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534907 5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534909 5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534912 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534914 5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534918 5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534920 5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534923 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534926 5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534928 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534931 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534933 5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534936 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:29:44.540914 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534938 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534941 5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534944 5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534946 5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534949 5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534951 5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534955 5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534959 5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534961 5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534964 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534967 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.534971 5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535449 5041 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535454 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535457 5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535460 5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535465 5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535468 5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535471 5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535473 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:29:44.541364 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535476 5041 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535479 5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535481 5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535483 5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535487 5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535490 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535492 5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535495 5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535499 5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535503 5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535506 5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535509 5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535511 5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535514 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535516 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535519 5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535521 5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535524 5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535527 5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:29:44.541896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535529 5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535532 5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535534 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535537 5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535539 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535542 5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535544 5041 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535547 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535550 5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535554 5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535557 5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535559 5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535561 5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535564 5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535567 5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535569 5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535572 5041 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535576 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535579 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:29:44.542354 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535581 5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535584 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535586 5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535589 5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535591 5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535594 5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535597 5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535600 5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535602 5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535605 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535607 5041 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535610 5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535614 5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535618 5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535620 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535624 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535627 5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535629 5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535632 5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:29:44.542818 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535634 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535637 5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535639 5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535644 5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535647 5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535649 5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535652 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535654 5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535657 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535675 5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535678 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535682 5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535684 5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535687 5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535689 5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535692 5041 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535694 5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535697 5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535700 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535702 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:29:44.543258 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535705 5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535707 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535710 5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535712 5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535715 5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535718 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535720 5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535723 5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535725 5041 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535728 5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535731 5041 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535733 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535736 5041 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535739 5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535741 5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535745 5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535749 5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535752 5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535754 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535757 5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.535759 5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:29:44.543760 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535853 5041 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535871 5041 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535882 5041 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535887 5041 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535891 5041 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535895 5041 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535899 5041 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535903 5041 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535906 5041 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535909 5041 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535912 5041 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535916 5041 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535919 5041 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535922 5041 flags.go:64] FLAG: --cgroup-root="" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535925 5041 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535928 5041 flags.go:64] FLAG: --client-ca-file="" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535930 5041 flags.go:64] FLAG: --cloud-provider="external" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535933 5041 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535938 5041 flags.go:64] FLAG: --cluster-domain="" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535941 5041 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535944 5041 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535947 5041 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535950 5041 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535954 5041 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 21:29:44.544266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535957 5041 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535960 5041 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535965 5041 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535968 5041 flags.go:64] FLAG: --contention-profiling="false" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535972 5041 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535975 5041 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535978 5041 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535981 5041 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535985 5041 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535988 5041 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535992 5041 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535995 5041 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.535998 5041 flags.go:64] FLAG: --enable-server="true" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536001 5041 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536006 5041 flags.go:64] FLAG: --event-burst="100" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536010 5041 flags.go:64] FLAG: --event-qps="50" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536013 5041 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536016 5041 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536019 5041 flags.go:64] FLAG: --eviction-hard="" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536023 5041 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536026 5041 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536029 5041 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536032 5041 flags.go:64] FLAG: --eviction-soft="" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536035 5041 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536038 5041 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 21:29:44.544869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536041 5041 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536044 5041 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536047 5041 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536049 5041 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536052 5041 flags.go:64] FLAG: --feature-gates="" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536056 5041 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536059 5041 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536062 5041 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536065 5041 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536069 5041 flags.go:64] FLAG: --healthz-port="10248" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536073 5041 flags.go:64] FLAG: --help="false" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536076 5041 flags.go:64] FLAG: --hostname-override="ip-10-0-131-253.ec2.internal" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536079 5041 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536082 5041 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536085 5041 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536088 5041 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536092 5041 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536095 5041 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536098 5041 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536101 5041 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536104 5041 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536107 5041 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536110 5041 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536113 5041 flags.go:64] FLAG: --kube-reserved="" Nov 03 21:29:44.545569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536116 5041 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536119 5041 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536122 5041 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536125 5041 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536128 5041 flags.go:64] FLAG: --lock-file="" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536131 5041 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536133 5041 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536136 5041 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536146 5041 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536149 5041 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536152 5041 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536155 5041 flags.go:64] FLAG: --logging-format="text" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536158 5041 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536161 5041 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536164 5041 flags.go:64] FLAG: --manifest-url="" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536167 5041 flags.go:64] FLAG: --manifest-url-header="" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536171 5041 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536176 5041 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536180 5041 flags.go:64] FLAG: --max-pods="110" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536183 5041 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536186 5041 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536189 5041 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536192 5041 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536195 5041 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536198 5041 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 21:29:44.546358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536202 5041 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536209 5041 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536212 5041 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536215 5041 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536218 5041 flags.go:64] FLAG: --pod-cidr="" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536221 5041 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536224 5041 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536227 5041 flags.go:64] FLAG: --pods-per-core="0" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536230 5041 flags.go:64] FLAG: --port="10250" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536233 5041 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536236 5041 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-091b66530eba67f85" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536239 5041 flags.go:64] FLAG: --qos-reserved="" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536241 5041 flags.go:64] FLAG: --read-only-port="10255" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536244 5041 flags.go:64] FLAG: --register-node="true" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536247 5041 flags.go:64] FLAG: --register-with-taints="" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536251 5041 flags.go:64] FLAG: --registry-burst="10" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536254 5041 flags.go:64] FLAG: --registry-qps="5" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536257 5041 flags.go:64] FLAG: --reserved-cpus="" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536259 5041 flags.go:64] FLAG: --reserved-memory="" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536263 5041 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536266 5041 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536269 5041 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536272 5041 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536275 5041 flags.go:64] FLAG: --runonce="false" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536279 5041 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536282 5041 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536285 5041 flags.go:64] FLAG: --seccomp-default="false" Nov 03 21:29:44.547407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536288 5041 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536291 5041 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536294 5041 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536298 5041 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536301 5041 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536305 5041 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536308 5041 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536311 5041 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536314 5041 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536317 5041 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536320 5041 flags.go:64] FLAG: --system-cgroups="" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536323 5041 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536328 5041 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536331 5041 flags.go:64] FLAG: --tls-cert-file="" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536334 5041 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536338 5041 flags.go:64] FLAG: --tls-min-version="" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536340 5041 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536343 5041 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536346 5041 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536349 5041 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536352 5041 flags.go:64] FLAG: --v="2" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536356 5041 flags.go:64] FLAG: --version="false" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536360 5041 flags.go:64] FLAG: --vmodule="" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536364 5041 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536367 5041 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 21:29:44.548345 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536493 5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536498 5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536500 5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536503 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536507 5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536510 5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536513 5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536515 5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536518 5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536521 5041 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536524 5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536527 5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536530 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536533 5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536536 5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536538 5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536541 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536543 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536546 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536549 5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:29:44.549103 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536551 5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536554 5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536556 5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536559 5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536562 5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536564 5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536567 5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536569 5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536572 5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536575 5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536577 5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536580 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536582 5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536585 5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536587 5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536590 5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536594 5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536596 5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536599 5041 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536601 5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:29:44.549624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536604 5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536607 5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536609 5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536613 5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536617 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536619 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536622 5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536624 5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536627 5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536630 5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536632 5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536635 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536637 5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536640 5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536643 5041 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536645 5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536648 5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536651 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536653 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:29:44.550374 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536656 5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536672 5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536675 5041 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536678 5041 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536680 5041 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536683 5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536685 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536688 5041 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536691 5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536695 5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536697 5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536700 5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536704 5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536708 5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536711 5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536713 5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536716 5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536726 5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536729 5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536732 5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:29:44.551083 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536735 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536737 5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536740 5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536742 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536745 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536747 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536750 5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536752 5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536755 5041 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536758 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536761 5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536765 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536768 5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536771 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536773 5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536776 5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536778 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536781 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536783 5041 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:29:44.551913 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536786 5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:29:44.552551 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536789 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:29:44.552551 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536793 5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:29:44.552551 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536796 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:29:44.552551 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536798 5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:29:44.552551 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536801 5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:29:44.552551 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536803 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:29:44.552551 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536806 5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:29:44.552551 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.536833 5041 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:29:44.552551 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536876 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:29:44.552551 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536880 5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:29:44.552551 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536884 5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:29:44.552551 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536887 5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:29:44.552551 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536890 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:29:44.552551 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536892 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:29:44.552551 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536895 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:29:44.552551 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536898 5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536900 5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536903 5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536906 5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536908 5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536911 5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536914 5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536916 5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536919 5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536921 5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536924 5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536927 5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536929 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536932 5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536935 5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536937 5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536940 5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536942 5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536946 5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536948 5041 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:29:44.553051 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536951 5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536953 5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536956 5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536958 5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536961 5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536964 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536966 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536970 5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536973 5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536976 5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536978 5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536981 5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536984 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536986 5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536989 5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536991 5041 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536994 5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536997 5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.536999 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:29:44.553646 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537002 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537005 5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537007 5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537010 5041 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537012 5041 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537015 5041 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537017 5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537020 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537022 5041 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537025 5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537027 5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537031 5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537033 5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537037 5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537041 5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537044 5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537047 5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537050 5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537053 5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537056 5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:29:44.554222 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537060 5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537062 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537065 5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537068 5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537071 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537074 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537076 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537078 5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537081 5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537083 5041 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537086 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537090 5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537094 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537096 5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537099 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537101 5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537104 5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537106 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537109 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:29:44.554817 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537111 5041 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537114 5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537116 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537118 5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537122 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537125 5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537127 5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537129 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537132 5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537134 5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537137 5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537139 5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537142 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537145 5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537147 5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537151 5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537154 5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537156 5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537159 5041 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537161 5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:29:44.555255 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.537164 5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.540829 5041 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.540845 5041 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.540862 5041 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.540867 5041 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540921 5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540926 5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540929 5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540932 5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540935 5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540937 5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540940 5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540943 5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540945 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540948 5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540950 5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540953 5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540956 5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540958 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:29:44.555727 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540961 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540963 5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540966 5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540969 5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540973 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540975 5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540978 5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540981 5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540983 5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540987 5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540991 5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540994 5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.540997 5041 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541000 5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541002 5041 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541006 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541008 5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541011 5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541015 5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:29:44.556226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541018 5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541021 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541023 5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541026 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541029 5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541032 5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541034 5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541037 5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541039 5041 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541042 5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541044 5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541050 5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541053 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541055 5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541058 5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541061 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541063 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541066 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541069 5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:29:44.556710 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541071 5041 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541074 5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541076 5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541079 5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541081 5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541084 5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541087 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541089 5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541092 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541095 5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541098 5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541101 5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541103 5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541105 5041 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541108 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541110 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541113 5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541115 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541118 5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:29:44.557144 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541121 5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541123 5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541126 5041 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541129 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541131 5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541135 5041 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541138 5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541140 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541143 5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541145 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541148 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541151 5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541153 5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541157 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541159 5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541161 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541164 5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541166 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541169 5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541172 5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:29:44.557577 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541175 5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:29:44.558048 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541177 5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:29:44.558048 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541180 5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:29:44.558048 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541182 5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:29:44.558048 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541185 5041 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:29:44.558048 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541188 5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:29:44.558048 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541190 5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:29:44.558048 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541193 5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:29:44.558048 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541196 5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:29:44.558048 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541198 5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:29:44.558048 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541201 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:29:44.558048 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541203 5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:29:44.558048 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541206 5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:29:44.558048 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541208 5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:29:44.558048 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541211 5041 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:29:44.558048 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.541237 5041 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:29:44.558048 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541387 5041 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541391 5041 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541395 5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541398 5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541401 5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541403 5041 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541405 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541409 5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541411 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541414 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541416 5041 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541419 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541422 5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541424 5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541427 5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541429 5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541432 5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541434 5041 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541437 5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541440 5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:29:44.558428 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541442 5041 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541445 5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541448 5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541450 5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541452 5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541455 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541457 5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541460 5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541463 5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541465 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541468 5041 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541470 5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541473 5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541475 5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541480 5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541483 5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541485 5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541488 5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541490 5041 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541493 5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:29:44.558969 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541495 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541498 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541501 5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541503 5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541506 5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541509 5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541511 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541514 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541518 5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541521 5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541525 5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541528 5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541530 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541533 5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541541 5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541544 5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541547 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541550 5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541552 5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:29:44.559607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541555 5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541558 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541560 5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541563 5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541565 5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541568 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541572 5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541577 5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541580 5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541582 5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541585 5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541588 5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541590 5041 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541593 5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541595 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541598 5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541600 5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541603 5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541606 5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:29:44.560197 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541608 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541610 5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541613 5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541615 5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541618 5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541621 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541623 5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541626 5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541628 5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541636 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541639 5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541642 5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541644 5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541647 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541650 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541652 5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541655 5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541657 5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541677 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:29:44.560845 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541680 5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:29:44.561400 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541684 5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:29:44.561400 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541687 5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:29:44.561400 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541689 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:29:44.561400 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541692 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:29:44.561400 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541695 5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:29:44.561400 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541697 5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:29:44.561400 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541706 5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:29:44.561400 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541709 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:29:44.561400 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.541732 5041 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 21:29:44.561400 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541782 5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 21:29:44.561400 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541786 5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 21:29:44.561400 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541789 5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 21:29:44.561400 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541792 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 21:29:44.561400 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541795 5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 21:29:44.561400 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541798 5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541800 5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541803 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541805 5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541807 5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541810 5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541813 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541817 5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541826 5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541832 5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541834 5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541837 5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541839 5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541842 5041 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541845 5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541847 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541853 5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541856 5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541858 5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 21:29:44.561959 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541862 5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541864 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541867 5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541869 5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541872 5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541875 5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541877 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541880 5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541883 5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541885 5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541888 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541890 5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541893 5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541895 5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541898 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541900 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541904 5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541907 5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541910 5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541912 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 21:29:44.562575 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541915 5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541917 5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541927 5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541930 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541932 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541934 5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541937 5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541939 5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541942 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541944 5041 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541947 5041 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541949 5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541953 5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541955 5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541957 5041 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541960 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541962 5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541965 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541967 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541969 5041 feature_gate.go:386] unrecognized feature gate: Example Nov 03 21:29:44.563110 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541972 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541974 5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541977 5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541979 5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541982 5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541984 5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541986 5041 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541989 5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541991 5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541994 5041 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541996 5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.541998 5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542001 5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542003 5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542006 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542015 5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542018 5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542021 5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542023 5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542026 5041 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 21:29:44.563597 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542028 5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542030 5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542033 5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542035 5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542038 5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542041 5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542043 5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542045 5041 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542048 5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542050 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542053 5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542055 5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542058 5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542060 5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542063 5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542065 5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542067 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542072 5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542075 5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 21:29:44.564066 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542078 5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 21:29:44.564626 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542080 5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 21:29:44.564626 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:44.542083 5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 21:29:44.564626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.542136 5041 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 21:29:44.564626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.543107 5041 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 21:29:44.564626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.543161 5041 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 21:29:44.564626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.543352 5041 server.go:1014] "Starting client certificate rotation" Nov 03 21:29:44.564626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.543548 5041 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 21:29:44.564626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.543652 5041 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 21:24:04 +0000 UTC" deadline="2028-07-08 01:46:42.98167381 +0000 UTC" Nov 03 21:29:44.564626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.543723 5041 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14692h16m58.437952685s" Nov 03 21:29:44.564626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.543982 5041 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 21:29:44.564626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.545141 5041 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 21:29:44.564626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.548322 5041 log.go:25] "Validated CRI v1 runtime API" Nov 03 21:29:44.564626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.552360 5041 log.go:25] "Validated CRI v1 image API" Nov 03 21:29:44.564626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.553439 5041 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 21:29:44.564626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.555513 5041 fs.go:135] Filesystem UUIDs: map[08600e6d-1f1b-4b84-aa43-a329ae1bb807:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 db89b5c9-e9d8-4271-a861-49f1a1341575:/dev/nvme0n1p3] Nov 03 21:29:44.565324 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.555528 5041 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07c53b8d563f478b77c7b6d440a45adb2b6e6cb433b25dd1d2ec2fad25ce7e2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07c53b8d563f478b77c7b6d440a45adb2b6e6cb433b25dd1d2ec2fad25ce7e2b/userdata/shm major:0 minor:322 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23b21928d442df069c187793509633da76f70ed3ff5f3f305c40e5b60e8e45f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23b21928d442df069c187793509633da76f70ed3ff5f3f305c40e5b60e8e45f2/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25b89988e6a8429f942020dc80ac09dcb82a3d8ccd1f067151e347a4cbd8e040/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25b89988e6a8429f942020dc80ac09dcb82a3d8ccd1f067151e347a4cbd8e040/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27382050184cb35e75ffbf18d7d41ecc433dfa43ae7e28e0f5b45306a34fda3f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27382050184cb35e75ffbf18d7d41ecc433dfa43ae7e28e0f5b45306a34fda3f/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ad3d8d6cdc0c805825cf44a5a711406ad557fbb9b6a79930ff3f0e0d742178f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ad3d8d6cdc0c805825cf44a5a711406ad557fbb9b6a79930ff3f0e0d742178f/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34f787a783a293e42de8ee6732122662b4dc128e93addec24f269ec8d904f624/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34f787a783a293e42de8ee6732122662b4dc128e93addec24f269ec8d904f624/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4869d9aea39cbfef1657dad47963eaed9304d5b8ff2b702f29dd9608d2591b5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4869d9aea39cbfef1657dad47963eaed9304d5b8ff2b702f29dd9608d2591b5c/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60cd9a89ee846b5d7caeff05ada37137631b71e8a299d752fff10bb925cee194/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60cd9a89ee846b5d7caeff05ada37137631b71e8a299d752fff10bb925cee194/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ba8f748bd56cf066e3fdc0b9b6ed58eec228f3c3edfd37afe21903f60c47644/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ba8f748bd56cf066e3fdc0b9b6ed58eec228f3c3edfd37afe21903f60c47644/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6be4ba4998986958b5c3e9d233284bf5f5541d8ed2bdd3f82aea70baeba5e738/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6be4ba4998986958b5c3e9d233284bf5f5541d8ed2bdd3f82aea70baeba5e738/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73b55eafe425c791994c391877e8fc3b4d2d910a6cb9455e82c5ab9db604ddea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73b55eafe425c791994c391877e8fc3b4d2d910a6cb9455e82c5ab9db604ddea/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/812f5bfd753120fc01dee8828ddc157907904f1ed900462e48a4d9126db9ab99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/812f5bfd753120fc01dee8828ddc157907904f1ed900462e48a4d9126db9ab99/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82d0d86428c0964b3f5d515c5687f67edddfd11e6fff4661560a7c4f27442124/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82d0d86428c0964b3f5d515c5687f67edddfd11e6fff4661560a7c4f27442124/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d22691ae89709aea408b145b52fd3ea07b74372958366203de5ae2e48bf3c41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d22691ae89709aea408b145b52fd3ea07b74372958366203de5ae2e48bf3c41/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99472fbf0a315300178bc050744a9e79958553dfdacf7183a13f57657a53a0e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99472fbf0a315300178bc050744a9e79958553dfdacf7183a13f57657a53a0e6/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e0e9f2295547f652724c09ece12e7743ce8ab9e9b7cfd99ceb5365cbe9b4d39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e0e9f2295547f652724c09ece12e7743ce8ab9e9b7cfd99ceb5365cbe9b4d39/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a35ad8d6727d685ad030cfba62fc44d2a35788cee701cb6b430e82037708f095/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a35ad8d6727d685ad030cfba62fc44d2a35788cee701cb6b430e82037708f095/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4d4e4ce5a8adb20888802a97ab5d6b6405195a50b99310b913350660a04735a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4d4e4ce5a8adb20888802a97ab5d6b6405195a50b99310b913350660a04735a/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5273bfd1b9cfa16e00110e1fae1986c072514af1ac6e63826edc3d0f4fc95a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5273bfd1b9cfa16e00110e1fae1986c072514af1ac6e63826edc3d0f4fc95a4/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6d02155668d60cb4a8d28bc7248b95465777dc8d30d1454526682d1c47663b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6d02155668d60cb4a8d28bc7248b95465777dc8d30d1454526682d1c47663b6/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bcf1d521ab66e7d26743f70d0164ebeb81aef32e0cea06893d82ceb83b8ce414/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bcf1d521ab66e7d26743f70d0164ebeb81aef32e0cea06893d82ceb83b8ce414/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd81ef1b72bc14e09804f0dd43588b3bcfdf4bb3fb36206e64c964e68eee2db6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd81ef1b72bc14e09804f0dd43588b3bcfdf4bb3fb36206e64c964e68eee2db6/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c051676d7f856b2e3930b0a6ef88bb2c7007d0086ab1088956499601724dd533/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c051676d7f856b2e3930b0a6ef88bb2c7007d0086ab1088956499601724dd533/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfc5b75bf4618eed3366761ac8dbc0b9d0201d7c47c1594e7accd9d259032a89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfc5b75bf4618eed3366761ac8dbc0b9d0201d7c47c1594e7accd9d259032a89/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d192bcfde6661cded8df6336b694e6d08725509d75b24e812a37ed3101ca6ceb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d192bcfde6661cded8df6336b694e6d08725509d75b24e812a37ed3101ca6ceb/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8e1d8d876cf1f51ccd3d3f29393e4bd36910f47c12d4a33fdeda7745f288abe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8e1d8d876cf1f51ccd3d3f29393e4bd36910f47c12d4a33fdeda7745f288abe/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0852ff0f-d078-452f-bb56-3c13512f57ce/volumes/kubernetes.io~projected/kube-api-access-xfcx5:{mountpoint:/var/lib/kubelet/pods/0852ff0f-d078-452f-bb56-3c13512f57ce/volumes/kubernetes.io~projected/kube-api-access-xfcx5 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0852ff0f-d078-452f-bb56-3c13512f57ce/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/0852ff0f-d078-452f-bb56-3c13512f57ce/volumes/kubernetes.io~secret/default-certificate major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0852ff0f-d078-452f-bb56-3c13512f57ce/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/0852ff0f-d078-452f-bb56-3c13512f57ce/volumes/kubernetes.io~secret/stats-auth major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d3ddfc6-6d13-4b06-983f-75cca78543ea/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0d3ddfc6-6d13-4b06-983f-75cca78543ea/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d3ddfc6-6d13-4b06-983f-75cca78543ea/volumes/kubernetes.io~projected/kube-api-access-cf9vr:{mountpoint:/var/lib/kubelet/pods/0d3ddfc6-6d13-4b06-983f-75cca78543ea/volumes/kubernetes.io~projected/kube-api-access-cf9vr major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d3ddfc6-6d13-4b06-983f-75cca78543ea/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0d3ddfc6-6d13-4b06-983f-75cca78543ea/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bc36693-102d-48e8-bb01-302b38055e05/volumes/kubernetes.io~projected/kube-api-access-x526l:{mountpoint:/var/lib/kubelet/pods/1bc36693-102d-48e8-bb01-302b38055e05/volumes/kubernetes.io~projected/kube-api-access-x526l major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f738a71-6d2b-4ade-ac92-990cf0a2b311/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1f738a71-6d2b-4ade-ac92-990cf0a2b311/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a406a0d-024a-411f-94c2-953a739c214f/volumes/kubernetes.io~projected/kube-api-access-c8bqr:{mountpoint:/var/lib/kubelet/pods/2a406a0d-024a-411f-94c2-953a739c214f/volumes/kubernetes.io~projected/kube-api-access-c8bqr major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a8bade0-e7b8-4d61-b71f-fdb4b2517320/volumes/kubernetes.io~projected/kube-api-access-kx68k:{mountpoint:/var/lib/kubelet/pods/2a8bade0-e7b8-4d61-b71f-fdb4b2517320/volumes/kubernetes.io~projected/kube-api-access-kx68k major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2/volumes/kubernetes.io~projected/kube-api-access-b9s65:{mountpoint:/var/lib/kubelet/pods/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2/volumes/kubernetes.io~projected/kube-api-access-b9s65 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37984c5f-7e58-4323-81f0-4bc7d485ab88/volumes/kubernetes.io~projected/kube-api-access-bmdxx:{mountpoint:/var/lib/kubelet/pods/37984c5f-7e58-4323-81f0-4bc7d485ab88/volumes/kubernetes.io~projected/kube-api-access-bmdxx major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37984c5f-7e58-4323-81f0-4bc7d485ab88/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/37984c5f-7e58-4323-81f0-4bc7d485ab88/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37a8184c-e9f5-4d96-a816-e0191c32ed06/volumes/kubernetes.io~projected/kube-api-access-6mddx:{mountpoint:/var/lib/kubelet/pods/37a8184c-e9f5-4d96-a816-e0191c32ed06/volumes/kubernetes.io~projected/kube-api-access-6mddx major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37a8184c-e9f5-4d96-a816-e0191c32ed06/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/37a8184c-e9f5-4d96-a816-e0191c32ed06/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d339ae6-a9e4-4258-8fe1-a393b943f953/volumes/kubernetes.io~projected/kube-api-access-k8fx4:{mountpoint:/var/lib/kubelet/pods/3d339ae6-a9e4-4258-8fe1-a393b943f953/volumes/kubernetes.io~projected/kube-api-access-k8fx4 major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d339ae6-a9e4-4258-8fe1-a393b943f953/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/3d339ae6-a9e4-4258-8fe1-a393b943f953/volumes/kubernetes.io~secret/signing-key major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43fb3f37-13b3-4eaa-9b92-6c343b1b2cef/volumes/kubernetes.io~projected/kube-api-access-8srsk:{mountpoint:/var/lib/kubelet/pods/43fb3f37-13b3-4eaa-9b92-6c343b1b2cef/volumes/kubernetes.io~projected/kube-api-access-8srsk major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57dae9a4-2abc-4829-9beb-f61383f46bfe/volumes/kubernetes.io~projected/kube-api-access-8pwnr:{mountpoint:/var/lib/kubelet/pods/57dae9a4-2abc-4829-9beb-f61383f46bfe/volumes/kubernetes.io~projected/kube-api-access-8pwnr major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58ec685b-55f0-42a2-9112-6fe8a813fd8a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/58ec685b-55f0-42a2-9112-6fe8a813fd8a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/710eeea5-8bbf-4b10-8167-45a61512ef7f/volumes/kubernetes.io~projected/kube-api-access-pwmwp:{mountpoint:/var/lib/kubelet/pods/710eeea5-8bbf-4b10-8167-45a61512ef7f/volumes/kubernetes.io~projected/kube-api-access-pwmwp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7/volumes/kubernetes.io~projected/kube-api-access-n2jlb:{mountpoint:/var/lib/kubelet/pods/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7/volumes/kubernetes.io~projected/kube-api-access-n2jlb major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a35bad8-8af4-40c5-aa2b-1a939cc72c5e/volumes/kubernetes.io~projected/kube-api-access-phnxv:{mountpoint:/var/lib/kubelet/pods/9a35bad8-8af4-40c5-aa2b-1a939cc72c5e/volumes/kubernetes.io~projected/kube-api-access-phnxv major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f173fd7-1ddb-4616-91b2-35a2bd6e0341/volumes/kubernetes.io~projected/kube-api-access-ct582:{mountpoint:/var/lib/kubelet/pods/9f173fd7-1ddb-4616-91b2-35a2bd6e0341/volumes/kubernetes.io~projected/kube-api-access-ct582 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6467adf-c26e-46ae-a212-98c9846bfe0e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/a6467adf-c26e-46ae-a212-98c9846bfe0e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6467adf-c26e-46ae-a212-98c9846bfe0e/volumes/kubernetes.io~projected/kube-api-access-sldfp:{mountpoint:/var/lib/kubelet/pods/a6467adf-c26e-46ae-a212-98c9846bfe0e/volumes/kubernetes.io~projected/kube-api-access-sldfp major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6467adf-c26e-46ae-a212-98c9846bfe0e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/a6467adf-c26e-46ae-a212-98c9846bfe0e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6467adf-c26e-46ae-a212-98c9846bfe0e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/a6467adf-c26e-46ae-a212-98c9846bfe0e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a734caac-db4c-4f6e-8dd0-415843ecd1b8/volumes/kubernetes.io~projected/kube-api-access-7nc6h:{mountpoint:/var/lib/kubelet/pods/a734caac-db4c-4f6e-8dd0-415843ecd1b8/volumes/kubernetes.io~projected/kube-api-access-7nc6h major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab012cc6-ec86-41af-b6a1-e9278469c6de/volumes/kubernetes.io~projected/kube-api-access-mmt5p:{mountpoint:/var/lib/kubelet/pods/ab012cc6-ec86-41af-b6a1-e9278469c6de/volumes/kubernetes.io~projected/kube-api-access-mmt5p major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab012cc6-ec86-41af-b6a1-e9278469c6de/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ab012cc6-ec86-41af-b6a1-e9278469c6de/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/add088ee-8843-4be8-ad89-f396481163c5/volumes/kubernetes.io~projected/kube-api-access-w6jjd:{mountpoint:/var/lib/kubelet/pods/add088ee-8843-4be8-ad89-f396481163c5/volumes/kubernetes.io~projected/kube-api-access-w6jjd major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b04c787c-4de0-483a-9f96-698fa046d04b/volumes/kubernetes.io~projected/kube-api-access-q9kvm:{mountpoint:/var/lib/kubelet/pods/b04c787c-4de0-483a-9f96-698fa046d04b/volumes/kubernetes.io~projected/kube-api-access-q9kvm major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba86e754-0fcc-46ab-a5e2-10609342eb13/volumes/kubernetes.io~projected/kube-api-access-5ktf5:{mountpoint:/var/lib/kubelet/pods/ba86e754-0fcc-46ab-a5e2-10609342eb13/volumes/kubernetes.io~projected/kube-api-access-5ktf5 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5013527-41c2-4355-9204-594fe21d3c8a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c5013527-41c2-4355-9204-594fe21d3c8a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5013527-41c2-4355-9204-594fe21d3c8a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c5013527-41c2-4355-9204-594fe21d3c8a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5013527-41c2-4355-9204-594fe21d3c8a/volumes/kubernetes.io~projected/kube-api-access-g5cfd:{mountpoint:/var/lib/kubelet/pods/c5013527-41c2-4355-9204-594fe21d3c8a/volumes/kubernetes.io~projected/kube-api-access-g5cfd major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/volumes/kubernetes.io~projected/kube-api-access-jbkft:{mountpoint:/var/lib/kubelet/pods/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/volumes/kubernetes.io~projected/kube-api-access-jbkft major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca003584-cd18-4740-a619-66c89853c773/volumes/kubernetes.io~projected/kube-api-access-d7dl8:{mountpoint:/var/lib/kubelet/pods/ca003584-cd18-4740-a619-66c89853c773/volumes/kubernetes.io~projected/kube-api-access-d7dl8 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb8cd0fc-653b-4f07-a501-f121ec73b7d5/volumes/kubernetes.io~projected/kube-api-access-k8ssv:{mountpoint:/var/lib/kubelet/pods/cb8cd0fc-653b-4f07-a501-f121ec73b7d5/volumes/kubernetes.io~projected/kube-api-access-k8ssv major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd1064c9-bade-4061-9b89-5d15e6a26a3d/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/cd1064c9-bade-4061-9b89-5d15e6a26a3d/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd1064c9-bade-4061-9b89-5d15e6a26a3d/volumes/kubernetes.io~projected/kube-api-access-mh5pc:{mountpoint:/var/lib/kubelet/pods/cd1064c9-bade-4061-9b89-5d15e6a26a3d/volumes/kubernetes.io~projected/kube-api-access-mh5pc major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd1064c9-bade-4061-9b89-5d15e6a26a3d/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/cd1064c9-bade-4061-9b89-5d15e6a26a3d/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd1064c9-bade-4061-9b89-5d15e6a26a3d/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/cd1064c9-bade-4061-9b89-5d15e6a26a3d/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d65ba521-c50d-47b4-82d9-1910dde4fbb1/volumes/kubernetes.io~projected/kube-api-access-rhwr2:{mountpoint:/var/lib/kubelet/pods/d65ba521-c50d-47b4-82d9-1910dde4fbb1/volumes/kubernetes.io~projected/kube-api-access-rhwr2 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8a307b9-8e43-4b73-941d-d5e2c035da14/volumes/kubernetes.io~projected/kube-api-access-dz5g7:{mountpoint:/var/lib/kubelet/pods/d8a307b9-8e43-4b73-941d-d5e2c035da14/volumes/kubernetes.io~projected/kube-api-access-dz5g7 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddbb8fca-c02a-4e5f-a31f-919194f46f12/volumes/kubernetes.io~projected/kube-api-access-fkjh7:{mountpoint:/var/lib/kubelet/pods/ddbb8fca-c02a-4e5f-a31f-919194f46f12/volumes/kubernetes.io~projected/kube-api-access-fkjh7 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efc25e42-1fb2-4fb4-b991-ce83b2d174d1/volumes/kubernetes.io~projected/kube-api-access-btpnr:{mountpoint:/var/lib/kubelet/pods/efc25e42-1fb2-4fb4-b991-ce83b2d174d1/volumes/kubernetes.io~projected/kube-api-access-btpnr major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb46e9cd-a477-4380-8d82-63a93d0f4099/volumes/kubernetes.io~projected/kube-api-access-mqf4v:{mountpoint:/var/lib/kubelet/pods/fb46e9cd-a477-4380-8d82-63a93d0f4099/volumes/kubernetes.io~projected/kube-api-access-mqf4v major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/eb689b82e6f619930d437b0857284b2934f408ff0450c9c5827ad092a92063f8/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/311e2173b904c506c0a911dfdfad3dc0203fd2058fb082916ac73866e40dc1c5/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3c4a184043d755b590cf8d765592f77d58db3f174ed3768814c4403e4735a5c1/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f622412311684e028b56971c8e2b1d81f8a64b346b06e447a8d48ddac0fee7f3/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/efd30c22ad2dd989d59a13814cfe4de2c66ad4534e92b2e8554cf046f51e65d2/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/50a65c171a80e4ae7d1e1d43a562fe90a980826077a7ee11b441b16aa7fab353/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6cdbfa67385fac7cfe426c450ee2cad1bfb9c9846f3c10d1839475b0129fefcc/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f605a94f1acd7ed86251d7e922a115f75199fe59bb21dc407e774e7c2aba27ac/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/21312687f9e148e8b20b57f798f979c367fcfc18271022cf71b046f869f92ba7/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/a459c5d344888c06f9a55fd2a24f383deb1a016a1149fe10fa2a1b28fd1f6707/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/9a97a32b70d53698af4a348b74697ece2c1fd565103bdcdf5cd70f627cc263a0/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/5ad9b30449afd1525b5555ea9ffac7742b42136fc5a8c162a2775ccf0a6da892/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0ec6a7b8e26b1be35f4a9a3668796f145931cd592e56a039ede54061ec468c0f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/26720e527ef9499fd84adfc39a37d09f430e2331e80c0c9cb7d0b5183b131b12/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/fc1d03b9b71f5518ea0127edb784f474603e89b09f002cf96799fa8cbb01d34c/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/ae2f0757c62e18e033e41dae15043af655dd5ae24e3d9b0877fe862946dfaea6/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/f80921f28cf26ec8e319e07b09b73bdd5333f0f3ad876afe36793d875bc6fd4b/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/7662a5fbd70ff018e4eb64a58d6a3ad9f3e5a692ab05af4491060f2018286de1/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/497643b86ebf799c81b545619b2fbfdfeb1842a6107d7df8d88fdfaa5ee4ab4f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/2340728a12252c259d1770051958c7fab34df16bc4fb0b1ef91cf0355f1458b9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ffbcaf28a02c376eb732bd1cade27b3477651d6bf9a2af6856dd07f79084d59a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/f9271850507769f78b29b67df16ac98c93545808df154e8283f4d132ff5ead6f/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/204357ec221784c9a52d2abdd587834b23d873aa734def2205f1a72f421a30ce/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/35cc1e8904bde205b636dd014dc3041246120986d50448313017e0e338b53ab2/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/4d0f1f3f19e497e308494c1ad9d8551270e7623b2e69a2bb4ac1098ebbd6e9f0/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/178d83fd327966e736ac2ac6b3c935a8d1aa06a12654256451388e7dede9a9d1/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/dcba5a17b4150c3b30f285ba9aadf99ebbe1ba09ff30a37f30da92a4c3858bb7/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/b4a117dab3f26665e705f0847f668cabfb73a1a0f372dfafa83e3d99b18f8852/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/f6082a17b4b2bc571d8234c41f92a71e39e3ad9a4dde766b5203585902448250/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/87cacde3b739214a90ab401825956b129cce64b5428ceab0f5e9696fe1791d93/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/1be16bdbddb489b81b3da0e8ba5ed0b2946355792256e42eff927d6e65f14b1b/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/dd4fc07b34fef8a172bf6a4f6a1bce347a1dd08a0e8d77cc273643eafb4042bc/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/9cf3b68393bb0f6d90e9c7bf31a8aaf57aacdc256ca3c4715f2219a720762d52/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/08e9c8283976a75f8a5697d6f38203604898137e88dc23073f9990fe358ab740/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/cba64202f9c622abe40634d00d3d6cd9e01080320dd32a4f8330ff559830ce35/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/a3633234423072a052bfe980d1f33e2e9f736276bbd2db9a3126926864ffbdde/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/5bcd459b750eafe102548937c79b50dc16ec2bb99db320ff2167c5d5c01d3219/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/26b31d6c80823c666162011d3e50423bcb2fdd5516e6c1f69998ed5585c1f3c0/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/2382ada598ddb0d94b418e39eeea44735fd2a0734366d98e67dae3faf920b6cf/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/362f8a8f6e91f38587e32ed6ec3096d9e21092a4c32a4b1a76d6823c01a4b728/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/792859fad9ec54d757b6bd22b33230d78108babfba575bb18cdcca03b4be4357/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/097003efbd4b7b2f17c8f8f41bb5943f4cc1ce5ee816b7224ba5827cdd9f6006/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/56492eb7a8bd1f142fe4d02122ade5d27b2637bb6fada4a9ae01a0062f83b90e/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/b1159649e8a2fa93b4f91c01fb0a353e03ebb5c543f06cc1546f7884207a343e/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/7aaf212c8d046cd11be7f14f5d7d43d850b7b9f60ef635163b6aab096b7b5a6f/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/87c6d064c2047cfb78bab0f3c2497b78cad12dcecb810bd4b9c859065f1757cf/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/0b749948f5a8b6f6bfdcd221d1d887a8542b5cb5fe1e8f3456c5e828d8ea595d/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/639b5fd835645f7e199c82e54b8867e6b8fa7c6e243b378eee37e2454cf523fa/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/bca9219e3188542b9fe90a5235bc18826cd056f496c194ee091b7e3b568d7ec3/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/7d8fc89efd1efa9229c33f58ba07c4a423e9320c122c56e27e9645bd02b4bf41/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/b8cbb2d9bca72ae0ace826c4192cc8c4dc873f610577b6abc1800e16ec723844/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/9e4f9ae2c1e6691040ba78b8c42d53b4333def42a2c78fa422e0eec46472510e/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/fe0800ed0058bb15bc63cc191e7d1f1d210cde004e69ff31eb8ffa98728e7777/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/b9eb7bc5ca85afc5be51285d049da5c4c05c979e50f1de86e72a877ef8c131e6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/60203a11f4780d82b315067f0d32e7da97375503c3aa56b3858c6601f147ce15/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/382813b04b1ac5e81d19ec77f5471fbdbac1b9be8c42163a87e5610c2d0508f4/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/f9b3c591f6352c5bc73ac0c1f24ac2c976d88fa2a930cce910a35499ddb083da/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/69dd7eee8fc299d1ee82e4bedcd7c3d42ef63c474b695b49ae708cecfa0293c7/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/93f677b85c4b9be0078d2d200e62e1ffbbac50cd4c0265e994cfe5effc3d70d8/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/a1304aeaabbf8e6a73b853fa2f1e449f53a5d313ad9ee74036b0a3031c941431/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/00ac56b7efddd1324241049a82c65a1a26fae340f7fcb4dc631c6b8b00643011/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c1c0e2763d5bdd408d350f84e02cb422a7e5f75dcb627f3c2e497a232588d8b5/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.563946 5041 manager.go:217] Machine: {Timestamp:2026-11-03 21:29:44.563474927 +0000 UTC m=+0.070703197 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3198972 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cdf649b406393e188eb5da64f21d7 SystemUUID:ec2cdf64-9b40-6393-e188-eb5da64f21d7 BootID:1aa59ee1-4161-4e23-9e03-c91e4c35c583 Filesystems:[{Device:/var/lib/kubelet/pods/add088ee-8843-4be8-ad89-f396481163c5/volumes/kubernetes.io~projected/kube-api-access-w6jjd DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/d65ba521-c50d-47b4-82d9-1910dde4fbb1/volumes/kubernetes.io~projected/kube-api-access-rhwr2 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2a406a0d-024a-411f-94c2-953a739c214f/volumes/kubernetes.io~projected/kube-api-access-c8bqr DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5273bfd1b9cfa16e00110e1fae1986c072514af1ac6e63826edc3d0f4fc95a4/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e0e9f2295547f652724c09ece12e7743ce8ab9e9b7cfd99ceb5365cbe9b4d39/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bcf1d521ab66e7d26743f70d0164ebeb81aef32e0cea06893d82ceb83b8ce414/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fb46e9cd-a477-4380-8d82-63a93d0f4099/volumes/kubernetes.io~projected/kube-api-access-mqf4v DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d3ddfc6-6d13-4b06-983f-75cca78543ea/volumes/kubernetes.io~projected/kube-api-access-cf9vr DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ab012cc6-ec86-41af-b6a1-e9278469c6de/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a6467adf-c26e-46ae-a212-98c9846bfe0e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7/volumes/kubernetes.io~projected/kube-api-access-n2jlb DeviceMajor:0 DeviceMinor:343 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6467adf-c26e-46ae-a212-98c9846bfe0e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ad3d8d6cdc0c805825cf44a5a711406ad557fbb9b6a79930ff3f0e0d742178f/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddbb8fca-c02a-4e5f-a31f-919194f46f12/volumes/kubernetes.io~projected/kube-api-access-fkjh7 DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d339ae6-a9e4-4258-8fe1-a393b943f953/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:346 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99472fbf0a315300178bc050744a9e79958553dfdacf7183a13f57657a53a0e6/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37984c5f-7e58-4323-81f0-4bc7d485ab88/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ab012cc6-ec86-41af-b6a1-e9278469c6de/volumes/kubernetes.io~projected/kube-api-access-mmt5p DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d192bcfde6661cded8df6336b694e6d08725509d75b24e812a37ed3101ca6ceb/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d339ae6-a9e4-4258-8fe1-a393b943f953/volumes/kubernetes.io~projected/kube-api-access-k8fx4 DeviceMajor:0 DeviceMinor:347 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5013527-41c2-4355-9204-594fe21d3c8a/volumes/kubernetes.io~projected/kube-api-access-g5cfd DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a734caac-db4c-4f6e-8dd0-415843ecd1b8/volumes/kubernetes.io~projected/kube-api-access-7nc6h DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60cd9a89ee846b5d7caeff05ada37137631b71e8a299d752fff10bb925cee194/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43fb3f37-13b3-4eaa-9b92-6c343b1b2cef/volumes/kubernetes.io~projected/kube-api-access-8srsk DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27382050184cb35e75ffbf18d7d41ecc433dfa43ae7e28e0f5b45306a34fda3f/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba86e754-0fcc-46ab-a5e2-10609342eb13/volumes/kubernetes.io~projected/kube-api-access-5ktf5 DeviceMajor:0 DeviceMinor:342 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cb8cd0fc-653b-4f07-a501-f121ec73b7d5/volumes/kubernetes.io~projected/kube-api-access-k8ssv DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58ec685b-55f0-42a2-9112-6fe8a813fd8a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:266 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a8bade0-e7b8-4d61-b71f-fdb4b2517320/volumes/kubernetes.io~projected/kube-api-access-kx68k DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37984c5f-7e58-4323-81f0-4bc7d485ab88/volumes/kubernetes.io~projected/kube-api-access-bmdxx DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b04c787c-4de0-483a-9f96-698fa046d04b/volumes/kubernetes.io~projected/kube-api-access-q9kvm DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd1064c9-bade-4061-9b89-5d15e6a26a3d/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ba8f748bd56cf066e3fdc0b9b6ed58eec228f3c3edfd37afe21903f60c47644/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a6467adf-c26e-46ae-a212-98c9846bfe0e/volumes/kubernetes.io~projected/kube-api-access-sldfp DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07c53b8d563f478b77c7b6d440a45adb2b6e6cb433b25dd1d2ec2fad25ce7e2b/userdata/shm DeviceMajor:0 DeviceMinor:322 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6be4ba4998986958b5c3e9d233284bf5f5541d8ed2bdd3f82aea70baeba5e738/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73b55eafe425c791994c391877e8fc3b4d2d910a6cb9455e82c5ab9db604ddea/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/710eeea5-8bbf-4b10-8167-45a61512ef7f/volumes/kubernetes.io~projected/kube-api-access-pwmwp DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0852ff0f-d078-452f-bb56-3c13512f57ce/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6467adf-c26e-46ae-a212-98c9846bfe0e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2/volumes/kubernetes.io~projected/kube-api-access-b9s65 DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d8a307b9-8e43-4b73-941d-d5e2c035da14/volumes/kubernetes.io~projected/kube-api-access-dz5g7 DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37a8184c-e9f5-4d96-a816-e0191c32ed06/volumes/kubernetes.io~projected/kube-api-access-6mddx DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd81ef1b72bc14e09804f0dd43588b3bcfdf4bb3fb36206e64c964e68eee2db6/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34f787a783a293e42de8ee6732122662b4dc128e93addec24f269ec8d904f624/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f738a71-6d2b-4ade-ac92-990cf0a2b311/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8e1d8d876cf1f51ccd3d3f29393e4bd36910f47c12d4a33fdeda7745f288abe/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d3ddfc6-6d13-4b06-983f-75cca78543ea/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0852ff0f-d078-452f-bb56-3c13512f57ce/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ca003584-cd18-4740-a619-66c89853c773/volumes/kubernetes.io~projected/kube-api-access-d7dl8 DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23b21928d442df069c187793509633da76f70ed3ff5f3f305c40e5b60e8e45f2/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9f173fd7-1ddb-4616-91b2-35a2bd6e0341/volumes/kubernetes.io~projected/kube-api-access-ct582 DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/812f5bfd753120fc01dee8828ddc157907904f1ed900462e48a4d9126db9ab99/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6d02155668d60cb4a8d28bc7248b95465777dc8d30d1454526682d1c47663b6/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37a8184c-e9f5-4d96-a816-e0191c32ed06/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/volumes/kubernetes.io~projected/kube-api-access-jbkft DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd1064c9-bade-4061-9b89-5d15e6a26a3d/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0852ff0f-d078-452f-bb56-3c13512f57ce/volumes/kubernetes.io~projected/kube-api-access-xfcx5 DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/efc25e42-1fb2-4fb4-b991-ce83b2d174d1/volumes/kubernetes.io~projected/kube-api-access-btpnr DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57dae9a4-2abc-4829-9beb-f61383f46bfe/volumes/kubernetes.io~projected/kube-api-access-8pwnr DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd1064c9-bade-4061-9b89-5d15e6a26a3d/volumes/kubernetes.io~projected/kube-api-access-mh5pc DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d3ddfc6-6d13-4b06-983f-75cca78543ea/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5013527-41c2-4355-9204-594fe21d3c8a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a35ad8d6727d685ad030cfba62fc44d2a35788cee701cb6b430e82037708f095/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfc5b75bf4618eed3366761ac8dbc0b9d0201d7c47c1594e7accd9d259032a89/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4869d9aea39cbfef1657dad47963eaed9304d5b8ff2b702f29dd9608d2591b5c/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5013527-41c2-4355-9204-594fe21d3c8a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a35bad8-8af4-40c5-aa2b-1a939cc72c5e/volumes/kubernetes.io~projected/kube-api-access-phnxv DeviceMajor:0 DeviceMinor:321 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25b89988e6a8429f942020dc80ac09dcb82a3d8ccd1f067151e347a4cbd8e040/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd1064c9-bade-4061-9b89-5d15e6a26a3d/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82d0d86428c0964b3f5d515c5687f67edddfd11e6fff4661560a7c4f27442124/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1bc36693-102d-48e8-bb01-302b38055e05/volumes/kubernetes.io~projected/kube-api-access-x526l DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d22691ae89709aea408b145b52fd3ea07b74372958366203de5ae2e48bf3c41/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4d4e4ce5a8adb20888802a97ab5d6b6405195a50b99310b913350660a04735a/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c051676d7f856b2e3930b0a6ef88bb2c7007d0086ab1088956499601724dd533/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:07c53b8d563f478 MacAddress:0a:eb:9e:d1:ba:d0 Speed:10000 Mtu:8901} {Name:25b89988e6a8429 MacAddress:f2:fa:85:1f:78:63 Speed:10000 Mtu:8901} {Name:27382050184cb35 MacAddress:b6:26:c0:27:7e:d0 Speed:10000 Mtu:8901} {Name:2ad3d8d6cdc0c80 MacAddress:e6:6e:e1:06:c4:46 Speed:10000 Mtu:8901} {Name:4869d9aea39cbfe MacAddress:92:57:15:8b:27:38 Speed:10000 Mtu:8901} {Name:60cd9a89ee846b5 MacAddress:c2:d5:6e:f5:7f:6e Speed:10000 Mtu:8901} {Name:812f5bfd753120f MacAddress:5e:7f:be:13:31:34 Speed:10000 Mtu:8901} {Name:82d0d86428c0964 MacAddress:9e:a5:00:90:99:c3 Speed:10000 Mtu:8901} {Name:9e0e9f2295547f6 MacAddress:16:7e:82:48:98:66 Speed:10000 Mtu:8901} {Name:a35ad8d6727d685 MacAddress:52:a1:42:2f:16:52 Speed:10000 Mtu:8901} {Name:a4d4e4ce5a8adb2 MacAddress:56:2c:fb:43:bc:09 Speed:10000 Mtu:8901} {Name:bcf1d521ab66e7d MacAddress:02:68:c7:7e:d4:c2 Speed:10000 Mtu:8901} {Name:bd81ef1b72bc14e MacAddress:1a:c8:62:19:ad:7d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c7:7d:b6:83 Speed:0 Mtu:9001} {Name:br-int MacAddress:82:bd:01:ab:9a:c8 Speed:0 Mtu:8901} {Name:c051676d7f856b2 MacAddress:c2:59:9e:79:48:52 Speed:10000 Mtu:8901} {Name:cfc5b75bf4618ee MacAddress:2a:91:25:c7:eb:a4 Speed:10000 Mtu:8901} {Name:d192bcfde6661cd MacAddress:fe:f8:ba:ff:51:18 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c7:7d:b6:83 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:62:e1:c5:36:be:ec Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:be:6a:db:da:95:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.564297 5041 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.564342 5041 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.564715 5041 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.564738 5041 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-253.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.564933 5041 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.564946 5041 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.564965 5041 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.564983 5041 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.565010 5041 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.565234 5041 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.565377 5041 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.565421 5041 kubelet.go:493] "Attempting to sync node with API server" Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.565438 5041 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.565451 5041 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.565463 5041 kubelet.go:405] "Adding apiserver pod source" Nov 03 21:29:44.565706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.565485 5041 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 21:29:44.566868 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.566851 5041 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 21:29:44.566868 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.566868 5041 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 21:29:44.567395 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.567375 5041 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 21:29:44.570767 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.570751 5041 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 21:29:44.570767 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.570765 5041 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 21:29:44.570884 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.570864 5041 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 21:29:44.571060 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.571048 5041 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 21:29:44.571097 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.571078 5041 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 21:29:44.571189 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.571179 5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 21:29:44.571221 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.571195 5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 21:29:44.571221 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.571202 5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 21:29:44.571221 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.571207 5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 21:29:44.571221 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.571214 5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 21:29:44.571328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.571223 5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 21:29:44.571328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.571232 5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 21:29:44.571328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.571240 5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 21:29:44.571328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.571247 5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 21:29:44.571328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.571254 5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 21:29:44.571328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.571262 5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 21:29:44.571328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.571271 5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 21:29:44.571328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.571298 5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 21:29:44.571328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.571307 5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 21:29:44.574744 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.574695 5041 server.go:1263] "Started kubelet" Nov 03 21:29:44.574872 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.574819 5041 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 21:29:44.574928 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.574893 5041 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 21:29:44.574928 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.574890 5041 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 21:29:44.575300 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.575281 5041 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 21:29:44.576016 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.576000 5041 server.go:317] "Adding debug handlers to kubelet server" Nov 03 21:29:44.577517 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.577501 5041 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 21:29:44.577604 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.577543 5041 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 21:29:44.577653 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.577587 5041 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 21:24:06 +0000 UTC" deadline="2028-05-05 02:09:34.242132545 +0000 UTC" Nov 03 21:29:44.577653 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.577622 5041 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13156h39m49.664513723s" Nov 03 21:29:44.577653 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:44.577624 5041 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-253.ec2.internal\" not found" Nov 03 21:29:44.577653 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.577643 5041 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 21:29:44.577653 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.577652 5041 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 21:29:44.577928 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.577765 5041 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 21:29:44.580343 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580296 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37984c5f-7e58-4323-81f0-4bc7d485ab88" volumeName="kubernetes.io/empty-dir/37984c5f-7e58-4323-81f0-4bc7d485ab88-snapshots" seLinuxMountContext="" Nov 03 21:29:44.580423 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580346 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd1064c9-bade-4061-9b89-5d15e6a26a3d" volumeName="kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-trusted-ca" seLinuxMountContext="" Nov 03 21:29:44.580423 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580361 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddbb8fca-c02a-4e5f-a31f-919194f46f12" volumeName="kubernetes.io/projected/ddbb8fca-c02a-4e5f-a31f-919194f46f12-kube-api-access-fkjh7" seLinuxMountContext="" Nov 03 21:29:44.580423 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580374 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb46e9cd-a477-4380-8d82-63a93d0f4099" volumeName="kubernetes.io/projected/fb46e9cd-a477-4380-8d82-63a93d0f4099-kube-api-access-mqf4v" seLinuxMountContext="" Nov 03 21:29:44.580423 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580388 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d3ddfc6-6d13-4b06-983f-75cca78543ea" volumeName="kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovnkube-config" seLinuxMountContext="" Nov 03 21:29:44.580423 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580400 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="add088ee-8843-4be8-ad89-f396481163c5" volumeName="kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-cni-binary-copy" seLinuxMountContext="" Nov 03 21:29:44.580423 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580414 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd1064c9-bade-4061-9b89-5d15e6a26a3d" volumeName="kubernetes.io/empty-dir/cd1064c9-bade-4061-9b89-5d15e6a26a3d-ca-trust-extracted" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580427 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd1064c9-bade-4061-9b89-5d15e6a26a3d" volumeName="kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-image-registry-private-configuration" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580442 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2" volumeName="kubernetes.io/projected/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2-kube-api-access-b9s65" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580453 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6467adf-c26e-46ae-a212-98c9846bfe0e" volumeName="kubernetes.io/empty-dir/a6467adf-c26e-46ae-a212-98c9846bfe0e-ca-trust-extracted" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580466 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab012cc6-ec86-41af-b6a1-e9278469c6de" volumeName="kubernetes.io/projected/ab012cc6-ec86-41af-b6a1-e9278469c6de-kube-api-access-mmt5p" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580476 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca003584-cd18-4740-a619-66c89853c773" volumeName="kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580486 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a406a0d-024a-411f-94c2-953a739c214f" volumeName="kubernetes.io/projected/2a406a0d-024a-411f-94c2-953a739c214f-kube-api-access-c8bqr" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580496 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab012cc6-ec86-41af-b6a1-e9278469c6de" volumeName="kubernetes.io/configmap/ab012cc6-ec86-41af-b6a1-e9278469c6de-config" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580510 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="add088ee-8843-4be8-ad89-f396481163c5" volumeName="kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580517 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b04c787c-4de0-483a-9f96-698fa046d04b" volumeName="kubernetes.io/configmap/b04c787c-4de0-483a-9f96-698fa046d04b-iptables-alerter-script" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580535 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5013527-41c2-4355-9204-594fe21d3c8a" volumeName="kubernetes.io/empty-dir/c5013527-41c2-4355-9204-594fe21d3c8a-etc-tuned" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580545 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a8184c-e9f5-4d96-a816-e0191c32ed06" volumeName="kubernetes.io/secret/37a8184c-e9f5-4d96-a816-e0191c32ed06-serving-cert" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580556 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a8bade0-e7b8-4d61-b71f-fdb4b2517320" volumeName="kubernetes.io/configmap/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-config-volume" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580570 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43fb3f37-13b3-4eaa-9b92-6c343b1b2cef" volumeName="kubernetes.io/projected/43fb3f37-13b3-4eaa-9b92-6c343b1b2cef-kube-api-access-8srsk" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580583 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57dae9a4-2abc-4829-9beb-f61383f46bfe" volumeName="kubernetes.io/configmap/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-config" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580591 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a35bad8-8af4-40c5-aa2b-1a939cc72c5e" volumeName="kubernetes.io/projected/9a35bad8-8af4-40c5-aa2b-1a939cc72c5e-kube-api-access-phnxv" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580598 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="add088ee-8843-4be8-ad89-f396481163c5" volumeName="kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580608 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d3ddfc6-6d13-4b06-983f-75cca78543ea" volumeName="kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovnkube-script-lib" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580619 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f738a71-6d2b-4ade-ac92-990cf0a2b311" volumeName="kubernetes.io/configmap/1f738a71-6d2b-4ade-ac92-990cf0a2b311-konnectivity-ca" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580626 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="710eeea5-8bbf-4b10-8167-45a61512ef7f" volumeName="kubernetes.io/projected/710eeea5-8bbf-4b10-8167-45a61512ef7f-kube-api-access-pwmwp" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580633 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f173fd7-1ddb-4616-91b2-35a2bd6e0341" volumeName="kubernetes.io/projected/9f173fd7-1ddb-4616-91b2-35a2bd6e0341-kube-api-access-ct582" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580640 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb46e9cd-a477-4380-8d82-63a93d0f4099" volumeName="kubernetes.io/configmap/fb46e9cd-a477-4380-8d82-63a93d0f4099-serviceca" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580649 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d3ddfc6-6d13-4b06-983f-75cca78543ea" volumeName="kubernetes.io/projected/0d3ddfc6-6d13-4b06-983f-75cca78543ea-kube-api-access-cf9vr" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580656 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37984c5f-7e58-4323-81f0-4bc7d485ab88" volumeName="kubernetes.io/secret/37984c5f-7e58-4323-81f0-4bc7d485ab88-serving-cert" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580683 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6467adf-c26e-46ae-a212-98c9846bfe0e" volumeName="kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-trusted-ca" seLinuxMountContext="" Nov 03 21:29:44.580703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580695 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6467adf-c26e-46ae-a212-98c9846bfe0e" volumeName="kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-installation-pull-secrets" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580719 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab012cc6-ec86-41af-b6a1-e9278469c6de" volumeName="kubernetes.io/secret/ab012cc6-ec86-41af-b6a1-e9278469c6de-serving-cert" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580731 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5013527-41c2-4355-9204-594fe21d3c8a" volumeName="kubernetes.io/empty-dir/c5013527-41c2-4355-9204-594fe21d3c8a-tmp" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580743 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a8ffb5d-6f44-499a-b1bc-1cf127993dc7" volumeName="kubernetes.io/projected/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-api-access-n2jlb" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580755 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6467adf-c26e-46ae-a212-98c9846bfe0e" volumeName="kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-image-registry-private-configuration" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580768 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37984c5f-7e58-4323-81f0-4bc7d485ab88" volumeName="kubernetes.io/projected/37984c5f-7e58-4323-81f0-4bc7d485ab88-kube-api-access-bmdxx" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580781 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d339ae6-a9e4-4258-8fe1-a393b943f953" volumeName="kubernetes.io/projected/3d339ae6-a9e4-4258-8fe1-a393b943f953-kube-api-access-k8fx4" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580794 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0852ff0f-d078-452f-bb56-3c13512f57ce" volumeName="kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-default-certificate" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580806 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37984c5f-7e58-4323-81f0-4bc7d485ab88" volumeName="kubernetes.io/configmap/37984c5f-7e58-4323-81f0-4bc7d485ab88-trusted-ca-bundle" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580818 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="710eeea5-8bbf-4b10-8167-45a61512ef7f" volumeName="kubernetes.io/configmap/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-daemon-config" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580830 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" volumeName="kubernetes.io/configmap/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-config" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580841 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d339ae6-a9e4-4258-8fe1-a393b943f953" volumeName="kubernetes.io/secret/3d339ae6-a9e4-4258-8fe1-a393b943f953-signing-key" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580849 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6467adf-c26e-46ae-a212-98c9846bfe0e" volumeName="kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-certificates" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580856 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6467adf-c26e-46ae-a212-98c9846bfe0e" volumeName="kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-bound-sa-token" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580863 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b04c787c-4de0-483a-9f96-698fa046d04b" volumeName="kubernetes.io/projected/b04c787c-4de0-483a-9f96-698fa046d04b-kube-api-access-q9kvm" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580869 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" volumeName="kubernetes.io/projected/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-kube-api-access-jbkft" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580876 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd1064c9-bade-4061-9b89-5d15e6a26a3d" volumeName="kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-certificates" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580883 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d65ba521-c50d-47b4-82d9-1910dde4fbb1" volumeName="kubernetes.io/empty-dir/d65ba521-c50d-47b4-82d9-1910dde4fbb1-tmp-dir" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580898 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0852ff0f-d078-452f-bb56-3c13512f57ce" volumeName="kubernetes.io/projected/0852ff0f-d078-452f-bb56-3c13512f57ce-kube-api-access-xfcx5" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580910 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="710eeea5-8bbf-4b10-8167-45a61512ef7f" volumeName="kubernetes.io/configmap/710eeea5-8bbf-4b10-8167-45a61512ef7f-cni-binary-copy" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580921 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d65ba521-c50d-47b4-82d9-1910dde4fbb1" volumeName="kubernetes.io/projected/d65ba521-c50d-47b4-82d9-1910dde4fbb1-kube-api-access-rhwr2" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580929 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6467adf-c26e-46ae-a212-98c9846bfe0e" volumeName="kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-kube-api-access-sldfp" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580936 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba86e754-0fcc-46ab-a5e2-10609342eb13" volumeName="kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-trusted-ca-bundle" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580943 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efc25e42-1fb2-4fb4-b991-ce83b2d174d1" volumeName="kubernetes.io/projected/efc25e42-1fb2-4fb4-b991-ce83b2d174d1-kube-api-access-btpnr" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580950 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a8184c-e9f5-4d96-a816-e0191c32ed06" volumeName="kubernetes.io/configmap/37a8184c-e9f5-4d96-a816-e0191c32ed06-config" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580963 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a8184c-e9f5-4d96-a816-e0191c32ed06" volumeName="kubernetes.io/projected/37a8184c-e9f5-4d96-a816-e0191c32ed06-kube-api-access-6mddx" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580971 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d3ddfc6-6d13-4b06-983f-75cca78543ea" volumeName="kubernetes.io/secret/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580981 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f738a71-6d2b-4ade-ac92-990cf0a2b311" volumeName="kubernetes.io/secret/1f738a71-6d2b-4ade-ac92-990cf0a2b311-agent-certs" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580989 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8a307b9-8e43-4b73-941d-d5e2c035da14" volumeName="kubernetes.io/projected/d8a307b9-8e43-4b73-941d-d5e2c035da14-kube-api-access-dz5g7" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.580995 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d3ddfc6-6d13-4b06-983f-75cca78543ea" volumeName="kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-env-overrides" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581001 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a8bade0-e7b8-4d61-b71f-fdb4b2517320" volumeName="kubernetes.io/empty-dir/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-tmp-dir" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581009 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a8bade0-e7b8-4d61-b71f-fdb4b2517320" volumeName="kubernetes.io/projected/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-kube-api-access-kx68k" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581016 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37984c5f-7e58-4323-81f0-4bc7d485ab88" volumeName="kubernetes.io/empty-dir/37984c5f-7e58-4323-81f0-4bc7d485ab88-tmp" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581023 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57dae9a4-2abc-4829-9beb-f61383f46bfe" volumeName="kubernetes.io/projected/57dae9a4-2abc-4829-9beb-f61383f46bfe-kube-api-access-8pwnr" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581031 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58ec685b-55f0-42a2-9112-6fe8a813fd8a" volumeName="kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581044 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a8ffb5d-6f44-499a-b1bc-1cf127993dc7" volumeName="kubernetes.io/empty-dir/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-data-volume" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581056 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9205351c-9ed0-4a10-8c52-0124c39a20fe" volumeName="kubernetes.io/configmap/9205351c-9ed0-4a10-8c52-0124c39a20fe-nginx-conf" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581068 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2" volumeName="kubernetes.io/secret/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2-cert" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581079 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d339ae6-a9e4-4258-8fe1-a393b943f953" volumeName="kubernetes.io/configmap/3d339ae6-a9e4-4258-8fe1-a393b943f953-signing-cabundle" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581092 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba86e754-0fcc-46ab-a5e2-10609342eb13" volumeName="kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-service-ca-bundle" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581104 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba86e754-0fcc-46ab-a5e2-10609342eb13" volumeName="kubernetes.io/projected/ba86e754-0fcc-46ab-a5e2-10609342eb13-kube-api-access-5ktf5" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581115 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5013527-41c2-4355-9204-594fe21d3c8a" volumeName="kubernetes.io/projected/c5013527-41c2-4355-9204-594fe21d3c8a-kube-api-access-g5cfd" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581127 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" volumeName="kubernetes.io/configmap/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-trusted-ca" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581139 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb8cd0fc-653b-4f07-a501-f121ec73b7d5" volumeName="kubernetes.io/projected/cb8cd0fc-653b-4f07-a501-f121ec73b7d5-kube-api-access-k8ssv" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581150 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd1064c9-bade-4061-9b89-5d15e6a26a3d" volumeName="kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-kube-api-access-mh5pc" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581161 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bc36693-102d-48e8-bb01-302b38055e05" volumeName="kubernetes.io/projected/1bc36693-102d-48e8-bb01-302b38055e05-kube-api-access-x526l" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581168 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a734caac-db4c-4f6e-8dd0-415843ecd1b8" volumeName="kubernetes.io/projected/a734caac-db4c-4f6e-8dd0-415843ecd1b8-kube-api-access-7nc6h" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581179 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="add088ee-8843-4be8-ad89-f396481163c5" volumeName="kubernetes.io/projected/add088ee-8843-4be8-ad89-f396481163c5-kube-api-access-w6jjd" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581186 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba86e754-0fcc-46ab-a5e2-10609342eb13" volumeName="kubernetes.io/empty-dir/ba86e754-0fcc-46ab-a5e2-10609342eb13-archives-path" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581193 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" volumeName="kubernetes.io/secret/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-serving-cert" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581199 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd1064c9-bade-4061-9b89-5d15e6a26a3d" volumeName="kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-bound-sa-token" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581206 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd1064c9-bade-4061-9b89-5d15e6a26a3d" volumeName="kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-installation-pull-secrets" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581213 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddbb8fca-c02a-4e5f-a31f-919194f46f12" volumeName="kubernetes.io/configmap/ddbb8fca-c02a-4e5f-a31f-919194f46f12-telemetry-config" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581219 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0852ff0f-d078-452f-bb56-3c13512f57ce" volumeName="kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-stats-auth" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581226 5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37984c5f-7e58-4323-81f0-4bc7d485ab88" volumeName="kubernetes.io/configmap/37984c5f-7e58-4323-81f0-4bc7d485ab88-service-ca-bundle" seLinuxMountContext="" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581234 5041 reconstruct.go:97] "Volume reconstruction finished" Nov 03 21:29:44.581738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.581239 5041 reconciler.go:29] "Reconciler: start to sync state" Nov 03 21:29:44.584011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.582133 5041 factory.go:55] Registering systemd factory Nov 03 21:29:44.584011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.582151 5041 factory.go:223] Registration of the systemd container factory successfully Nov 03 21:29:44.584011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.582386 5041 factory.go:153] Registering CRI-O factory Nov 03 21:29:44.584011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.582398 5041 factory.go:223] Registration of the crio container factory successfully Nov 03 21:29:44.584011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.582453 5041 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 21:29:44.584011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.582473 5041 factory.go:103] Registering Raw factory Nov 03 21:29:44.584011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.582488 5041 manager.go:1199] Started watching for new ooms in manager Nov 03 21:29:44.584011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.582837 5041 manager.go:319] Starting recovery of all containers Nov 03 21:29:44.582049 ip-10-0-131-253 systemd[1]: Started Kubernetes Kubelet. Nov 03 21:29:44.604940 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.604915 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/1.log" Nov 03 21:29:44.605936 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.605913 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/1.log" Nov 03 21:29:44.614682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.611818 5041 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 21:29:44.614682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.613210 5041 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 21:29:44.614682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.613231 5041 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 21:29:44.614682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.613254 5041 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 21:29:44.614682 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:44.613301 5041 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 21:29:44.629466 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.629435 5041 generic.go:356] "Generic (PLEG): container finished" podID="d42cfc86d4650e909d7de42d9326db55" containerID="68d436197fa23503099d082adec91ff654e3de302580cae3bd2eb540204197e1" exitCode=0 Nov 03 21:29:44.631057 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.631037 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/1.log" Nov 03 21:29:44.631161 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.631062 5041 generic.go:356] "Generic (PLEG): container finished" podID="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" containerID="5b4e56a0a125fb45430d89a491d2d78b22f384f0c1b89cd5dec41837a5111987" exitCode=255 Nov 03 21:29:44.631687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.631635 5041 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 21:29:44.631974 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.631799 5041 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:29:44.632061 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.632020 5041 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:29:44.637269 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.637244 5041 generic.go:356] "Generic (PLEG): container finished" podID="add088ee-8843-4be8-ad89-f396481163c5" containerID="db0dc9577da37c57b1a273822b0821bbbb08c018983f1360f4cb9bef90293490" exitCode=0 Nov 03 21:29:44.637269 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.637266 5041 generic.go:356] "Generic (PLEG): container finished" podID="add088ee-8843-4be8-ad89-f396481163c5" containerID="13ba642705953d82f39c75af97844e8ff9b5dafba7af951d60642f8b91fd5e76" exitCode=0 Nov 03 21:29:44.637378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.637276 5041 generic.go:356] "Generic (PLEG): container finished" podID="add088ee-8843-4be8-ad89-f396481163c5" containerID="47c5c03cbbf78059cb71d976ed066ca70fab06083d088f691a2c7ed58a0599c8" exitCode=0 Nov 03 21:29:44.637378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.637289 5041 generic.go:356] "Generic (PLEG): container finished" podID="add088ee-8843-4be8-ad89-f396481163c5" containerID="b2b4cf4d77b10dfacee0863c3017dc40d76cf0b33ebde177673c01eb7ac6628a" exitCode=0 Nov 03 21:29:44.637378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.637296 5041 generic.go:356] "Generic (PLEG): container finished" podID="add088ee-8843-4be8-ad89-f396481163c5" containerID="079082d63f644f8f917099442a8bfc9101ef9d9a63a74a1eac94dd3429e2d702" exitCode=0 Nov 03 21:29:44.637378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.637305 5041 generic.go:356] "Generic (PLEG): container finished" podID="add088ee-8843-4be8-ad89-f396481163c5" containerID="35910c1394eb957219207d7a4230dca0cf6bc97ce0474eaaa92473e10edf36f4" exitCode=0 Nov 03 21:29:44.646467 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.646426 5041 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:29:44.646935 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.646912 5041 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 21:29:44.692967 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.692908 5041 manager.go:324] Recovery completed Nov 03 21:29:44.713748 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.713717 5041 cpu_manager.go:226] "Starting" policy="none" Nov 03 21:29:44.713748 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.713744 5041 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 21:29:44.713927 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.713766 5041 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 21:29:44.713976 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.713963 5041 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 21:29:44.714023 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.713979 5041 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 21:29:44.714023 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.714005 5041 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 21:29:44.714023 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.714015 5041 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 21:29:44.714023 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.714023 5041 policy_none.go:50] "Start" Nov 03 21:29:44.714216 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.714061 5041 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 21:29:44.714216 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.714075 5041 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 21:29:44.714323 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.714308 5041 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 21:29:44.714376 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.714327 5041 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 21:29:44.714376 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.714337 5041 policy_none.go:44] "Start" Nov 03 21:29:44.714956 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:44.714928 5041 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 21:29:44.719455 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.719428 5041 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 21:29:44.724685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.722447 5041 manager.go:341] "Starting Device Plugin manager" Nov 03 21:29:44.724685 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:44.723139 5041 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 21:29:44.724685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.723156 5041 server.go:87] "Starting device plugin registration server" Nov 03 21:29:44.724685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.723453 5041 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 21:29:44.724685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.723466 5041 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 21:29:44.724685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.723586 5041 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 21:29:44.724685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.723703 5041 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 21:29:44.724685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.723741 5041 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 21:29:44.724685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.723748 5041 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 21:29:44.724685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.723789 5041 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T21:29:44.723731263Z","UUID":"11ad9534-f002-4b2f-83f2-f6ed914c0f76","Handler":null,"Name":"","Endpoint":""} Nov 03 21:29:44.725228 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.725216 5041 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 21:29:44.725270 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.725236 5041 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 21:29:44.726425 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:44.725998 5041 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 21:29:44.824021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.823987 5041 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 21:29:44.825223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.825205 5041 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientMemory" Nov 03 21:29:44.825301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.825235 5041 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 21:29:44.825301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.825245 5041 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-253.ec2.internal" event="NodeHasSufficientPID" Nov 03 21:29:44.825301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.825287 5041 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-253.ec2.internal" Nov 03 21:29:44.861574 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.861550 5041 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-253.ec2.internal" Nov 03 21:29:44.861694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.861614 5041 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-253.ec2.internal" Nov 03 21:29:44.915250 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.915204 5041 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal"] Nov 03 21:29:44.915855 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.915819 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" event={"ID":"9dff6629a956b5c1c96ca015020c4e26","Type":"ContainerStarted","Data":"1dfea53a3b721e82670647ce2da5f216091e49eb2a9c498d6a63195349fb1e99"} Nov 03 21:29:44.915980 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.915861 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" event={"ID":"9dff6629a956b5c1c96ca015020c4e26","Type":"ContainerStarted","Data":"6ba8f748bd56cf066e3fdc0b9b6ed58eec228f3c3edfd37afe21903f60c47644"} Nov 03 21:29:44.915980 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.915881 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" event={"ID":"d42cfc86d4650e909d7de42d9326db55","Type":"ContainerStarted","Data":"372f8cf78df34651a8fa7f2fe91692c75cebd18e33f4ab9ac3d4ee8ddc408ae6"} Nov 03 21:29:44.915980 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.915897 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" event={"ID":"d42cfc86d4650e909d7de42d9326db55","Type":"ContainerDied","Data":"68d436197fa23503099d082adec91ff654e3de302580cae3bd2eb540204197e1"} Nov 03 21:29:44.915980 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.915917 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" event={"ID":"d42cfc86d4650e909d7de42d9326db55","Type":"ContainerStarted","Data":"99472fbf0a315300178bc050744a9e79958553dfdacf7183a13f57657a53a0e6"} Nov 03 21:29:44.915980 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.915951 5041 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" Nov 03 21:29:44.916159 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.916025 5041 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:44.941578 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.941557 5041 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 21:29:44.941701 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:44.941598 5041 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-253.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" Nov 03 21:29:44.941992 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:44.941978 5041 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 21:29:44.942022 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:44.942014 5041 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:45.033337 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.033306 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d42cfc86d4650e909d7de42d9326db55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"d42cfc86d4650e909d7de42d9326db55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:45.033463 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.033339 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d42cfc86d4650e909d7de42d9326db55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"d42cfc86d4650e909d7de42d9326db55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:45.033463 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.033362 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9dff6629a956b5c1c96ca015020c4e26-config\") pod \"kube-apiserver-proxy-ip-10-0-131-253.ec2.internal\" (UID: \"9dff6629a956b5c1c96ca015020c4e26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" Nov 03 21:29:45.134398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.134357 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d42cfc86d4650e909d7de42d9326db55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"d42cfc86d4650e909d7de42d9326db55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:45.134553 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.134407 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d42cfc86d4650e909d7de42d9326db55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"d42cfc86d4650e909d7de42d9326db55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:45.134553 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.134432 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9dff6629a956b5c1c96ca015020c4e26-config\") pod \"kube-apiserver-proxy-ip-10-0-131-253.ec2.internal\" (UID: \"9dff6629a956b5c1c96ca015020c4e26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" Nov 03 21:29:45.134553 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.134444 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d42cfc86d4650e909d7de42d9326db55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"d42cfc86d4650e909d7de42d9326db55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:45.134553 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.134476 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9dff6629a956b5c1c96ca015020c4e26-config\") pod \"kube-apiserver-proxy-ip-10-0-131-253.ec2.internal\" (UID: \"9dff6629a956b5c1c96ca015020c4e26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" Nov 03 21:29:45.134553 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.134520 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d42cfc86d4650e909d7de42d9326db55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" (UID: \"d42cfc86d4650e909d7de42d9326db55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:45.565966 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.565937 5041 apiserver.go:52] "Watching apiserver" Nov 03 21:29:45.609689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.609647 5041 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 21:29:45.610386 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.610355 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-qvv5p","openshift-image-registry/node-ca-rwd6w","openshift-ingress-canary/ingress-canary-x4wh2","openshift-insights/periodic-gathering-dflhb-2sf79","openshift-kube-storage-version-migrator/migrator-66df5d496b-tggjh","openshift-network-console/networking-console-plugin-86c478bb88-w682j","kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal","openshift-insights/insights-operator-7c4986d449-wxk7n","kube-system/konnectivity-agent-vh4hd","openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f","openshift-dns/dns-default-7t4b8","openshift-kube-storage-version-migrator/migrator-66df5d496b-h2s5x","openshift-network-diagnostics/network-check-target-pllqd","openshift-multus/multus-gdt8s","openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr","kube-system/kas-connection-checker-685895c7c9-mq2cr","openshift-cluster-node-tuning-operator/tuned-cvc9c","openshift-ovn-kubernetes/ovnkube-node-ltgjc","openshift-dns/node-resolver-5wq87","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zthx8","openshift-console-operator/console-operator-7cd56b8dbc-ltzz4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv","openshift-insights/insights-runtime-extractor-plrfk","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb","openshift-network-operator/iptables-alerter-rc5dj","openshift-network-diagnostics/network-check-source-6f78d59f99-m2hhx","openshift-image-registry/image-registry-7fd7fc569c-h759l","openshift-service-ca/service-ca-5fffc97bdf-rg8rf","kube-system/kas-connection-checker-685895c7c9-pp685","openshift-image-registry/image-registry-7bf564b6d-c6gsv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal","kube-system/kas-connection-checker-685895c7c9-7f6xj","kube-system/global-pull-secret-syncer-z5jjd","openshift-ingress/router-default-9f5c5b5d5-hpzcq","openshift-multus/multus-additional-cni-plugins-2lwrg"] Nov 03 21:29:45.611498 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.611477 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.612076 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.612047 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:45.612323 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.612306 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:45.613507 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.613487 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:45.614578 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.614562 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:45.623202 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.623185 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:45.624098 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.624082 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.625072 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.625052 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:45.626395 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.626375 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.626395 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.626386 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.627972 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.627954 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:45.628400 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.628385 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 21:29:45.628467 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.628412 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 21:29:45.628467 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.628453 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 21:29:45.628694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.628684 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 21:29:45.628837 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.628825 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 21:29:45.629397 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.629383 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 21:29:45.629397 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.629393 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 21:29:45.629507 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.629441 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 21:29:45.629507 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.629456 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 21:29:45.629507 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.629454 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 21:29:45.629507 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.629474 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 21:29:45.629622 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.629509 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 21:29:45.629622 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.629590 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 21:29:45.629728 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.629715 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 21:29:45.629876 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.629861 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.630441 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.630424 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.630482 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.630424 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.630482 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.630432 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 21:29:45.630482 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.630471 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.630695 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.630682 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.630774 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.630761 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 21:29:45.630774 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.630770 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.630989 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.630976 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 21:29:45.631045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.630992 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.631248 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.631233 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 21:29:45.631294 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.631259 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.631294 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.631275 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 21:29:45.631381 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.631313 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.631381 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.631320 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.631381 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.631325 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.631381 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.631361 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 21:29:45.631381 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.631379 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.631617 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.631359 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.631723 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.631707 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.631795 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.631741 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.632056 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.632041 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.632125 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.632088 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.644967 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.644944 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 21:29:45.645310 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.645289 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 21:29:45.645479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.645464 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bxcrl\"" Nov 03 21:29:45.645547 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.645508 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 21:29:45.645599 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.645554 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 21:29:45.645599 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.645561 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.645722 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.645636 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 21:29:45.645776 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.645727 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.645776 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.645737 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 21:29:45.645862 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.645804 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.645909 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.645888 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x8jbx\"" Nov 03 21:29:45.645954 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.645914 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lj4ds\"" Nov 03 21:29:45.646037 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646021 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 21:29:45.646162 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646147 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4tzc5\"" Nov 03 21:29:45.646270 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646255 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.646388 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646373 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vzdjb\"" Nov 03 21:29:45.646445 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646424 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.646494 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646465 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 21:29:45.646547 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646529 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.646609 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646594 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mhg7x\"" Nov 03 21:29:45.646657 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646616 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.646729 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646694 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 21:29:45.646797 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646779 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 21:29:45.646840 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646804 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.646840 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646818 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 21:29:45.646932 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646836 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6tvtx\"" Nov 03 21:29:45.646932 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646873 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 21:29:45.646932 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646931 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.647056 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.646937 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-qmg6s\"" Nov 03 21:29:45.647056 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647009 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qcvh8\"" Nov 03 21:29:45.647166 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647149 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w4krc\"" Nov 03 21:29:45.647235 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647214 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 21:29:45.647395 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647380 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 21:29:45.647502 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647481 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tnnmd\"" Nov 03 21:29:45.648033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647827 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 21:29:45.648033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647845 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 21:29:45.648033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647853 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 21:29:45.648033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647867 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.648033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647871 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.648033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647918 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 21:29:45.648033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647942 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 21:29:45.648033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647971 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nz5lt\"" Nov 03 21:29:45.648033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647949 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.648033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647955 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.648033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647986 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.648033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.647957 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 21:29:45.648033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648001 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d5mwz\"" Nov 03 21:29:45.648033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648017 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rf9jq\"" Nov 03 21:29:45.648033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648026 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 21:29:45.648647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648071 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qc6g7\"" Nov 03 21:29:45.648647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648114 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.648647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648138 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 21:29:45.648647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648168 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.648647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648197 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xgwf4\"" Nov 03 21:29:45.648647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648245 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 21:29:45.648647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648262 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.648647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648304 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.648647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648305 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 21:29:45.648647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648351 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.648647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648408 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 21:29:45.648647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648408 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-th7gv\"" Nov 03 21:29:45.648647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648459 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 21:29:45.648647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648515 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2kjq7\"" Nov 03 21:29:45.648647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648565 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 21:29:45.648647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648600 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 21:29:45.649126 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648689 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6tkv6\"" Nov 03 21:29:45.649126 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.648738 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fv5g9\"" Nov 03 21:29:45.677329 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.677315 5041 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" Nov 03 21:29:45.677451 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.677439 5041 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:45.680382 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.680367 5041 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 21:29:45.699111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.699095 5041 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 21:29:45.699161 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.699131 5041 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-253.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-253.ec2.internal" Nov 03 21:29:45.699392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.699378 5041 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 21:29:45.699422 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.699413 5041 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-253.ec2.internal" Nov 03 21:29:45.738647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.738620 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-daemon-config\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.738647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.738650 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-slash\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.738780 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.738678 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf9vr\" (UniqueName: \"kubernetes.io/projected/0d3ddfc6-6d13-4b06-983f-75cca78543ea-kube-api-access-cf9vr\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.738820 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.738803 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ba86e754-0fcc-46ab-a5e2-10609342eb13-archives-path\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:45.738849 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.738838 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d65ba521-c50d-47b4-82d9-1910dde4fbb1-hosts-file\") pod \"node-resolver-5wq87\" (UID: \"d65ba521-c50d-47b4-82d9-1910dde4fbb1\") " pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:45.738904 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.738865 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.738960 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.738939 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ba86e754-0fcc-46ab-a5e2-10609342eb13-archives-path\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:45.738991 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.738979 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-daemon-config\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.739026 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.738941 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.739056 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739025 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-lib-modules\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.739056 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739045 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-tmp-dir\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:45.739126 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739105 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/710eeea5-8bbf-4b10-8167-45a61512ef7f-cni-binary-copy\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.739155 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739127 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-tmp-dir\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:45.739191 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739157 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-etc-kubernetes\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.739191 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739166 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.739191 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739184 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-etc-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.739283 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739206 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-config\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:45.739283 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739223 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz5g7\" (UniqueName: \"kubernetes.io/projected/d8a307b9-8e43-4b73-941d-d5e2c035da14-kube-api-access-dz5g7\") pod \"kas-connection-checker-685895c7c9-7f6xj\" (UID: \"d8a307b9-8e43-4b73-941d-d5e2c035da14\") " pod="kube-system/kas-connection-checker-685895c7c9-7f6xj" Nov 03 21:29:45.739283 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739252 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.739370 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739299 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/710eeea5-8bbf-4b10-8167-45a61512ef7f-cni-binary-copy\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.739370 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739336 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-service-ca-bundle\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:45.739370 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739367 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-trusted-ca\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.739472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739382 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/37984c5f-7e58-4323-81f0-4bc7d485ab88-snapshots\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:45.739472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739394 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-config\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:45.739472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739438 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37984c5f-7e58-4323-81f0-4bc7d485ab88-service-ca-bundle\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:45.739472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739461 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/37984c5f-7e58-4323-81f0-4bc7d485ab88-snapshots\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:45.739651 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739476 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb46e9cd-a477-4380-8d82-63a93d0f4099-host\") pod \"node-ca-rwd6w\" (UID: \"fb46e9cd-a477-4380-8d82-63a93d0f4099\") " pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:45.739651 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739499 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-service-ca-bundle\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:45.739651 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739530 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovnkube-script-lib\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.739651 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739559 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37984c5f-7e58-4323-81f0-4bc7d485ab88-service-ca-bundle\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:45.739651 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739564 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-trusted-ca\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:45.739651 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739597 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-multus-certs\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.740013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739653 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-socket-dir-parent\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.740013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739696 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmdxx\" (UniqueName: \"kubernetes.io/projected/37984c5f-7e58-4323-81f0-4bc7d485ab88-kube-api-access-bmdxx\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:45.740013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739722 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-ovn\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.740013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739740 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37a8184c-e9f5-4d96-a816-e0191c32ed06-config\") pod \"service-ca-operator-544d749c8c-r9qwk\" (UID: \"37a8184c-e9f5-4d96-a816-e0191c32ed06\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:45.740013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739793 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b04c787c-4de0-483a-9f96-698fa046d04b-iptables-alerter-script\") pod \"iptables-alerter-rc5dj\" (UID: \"b04c787c-4de0-483a-9f96-698fa046d04b\") " pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:45.740013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739818 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:45.740013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739820 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-trusted-ca\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.740013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739839 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovnkube-script-lib\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.740013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739858 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bqr\" (UniqueName: \"kubernetes.io/projected/2a406a0d-024a-411f-94c2-953a739c214f-kube-api-access-c8bqr\") pod \"network-metrics-daemon-qvv5p\" (UID: \"2a406a0d-024a-411f-94c2-953a739c214f\") " pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:45.740013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739895 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-crio-socket\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:45.740013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739908 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-trusted-ca\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:45.740013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739920 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37984c5f-7e58-4323-81f0-4bc7d485ab88-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:45.740013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739943 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37a8184c-e9f5-4d96-a816-e0191c32ed06-config\") pod \"service-ca-operator-544d749c8c-r9qwk\" (UID: \"37a8184c-e9f5-4d96-a816-e0191c32ed06\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:45.740013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.739964 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-systemd\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.740013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740005 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-trusted-ca-bundle\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740038 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-run\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740096 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-kubelet\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740119 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nc6h\" (UniqueName: \"kubernetes.io/projected/a734caac-db4c-4f6e-8dd0-415843ecd1b8-kube-api-access-7nc6h\") pod \"kas-connection-checker-685895c7c9-mq2cr\" (UID: \"a734caac-db4c-4f6e-8dd0-415843ecd1b8\") " pod="kube-system/kas-connection-checker-685895c7c9-mq2cr" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740137 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-bound-sa-token\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740170 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b04c787c-4de0-483a-9f96-698fa046d04b-iptables-alerter-script\") pod \"iptables-alerter-rc5dj\" (UID: \"b04c787c-4de0-483a-9f96-698fa046d04b\") " pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740196 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740225 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysctl-d\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740260 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-var-lib-kubelet\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740327 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab012cc6-ec86-41af-b6a1-e9278469c6de-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hplgb\" (UID: \"ab012cc6-ec86-41af-b6a1-e9278469c6de\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740354 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37984c5f-7e58-4323-81f0-4bc7d485ab88-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740358 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx68k\" (UniqueName: \"kubernetes.io/projected/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-kube-api-access-kx68k\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740405 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ddbb8fca-c02a-4e5f-a31f-919194f46f12-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740428 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-trusted-ca-bundle\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740422 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-cni-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740472 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-cnibin\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740494 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btpnr\" (UniqueName: \"kubernetes.io/projected/efc25e42-1fb2-4fb4-b991-ce83b2d174d1-kube-api-access-btpnr\") pod \"kas-connection-checker-685895c7c9-pp685\" (UID: \"efc25e42-1fb2-4fb4-b991-ce83b2d174d1\") " pod="kube-system/kas-connection-checker-685895c7c9-pp685" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740508 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab012cc6-ec86-41af-b6a1-e9278469c6de-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hplgb\" (UID: \"ab012cc6-ec86-41af-b6a1-e9278469c6de\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740516 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740616 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ddbb8fca-c02a-4e5f-a31f-919194f46f12-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:45.740654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740631 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740697 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mddx\" (UniqueName: \"kubernetes.io/projected/37a8184c-e9f5-4d96-a816-e0191c32ed06-kube-api-access-6mddx\") pod \"service-ca-operator-544d749c8c-r9qwk\" (UID: \"37a8184c-e9f5-4d96-a816-e0191c32ed06\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740722 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ssv\" (UniqueName: \"kubernetes.io/projected/cb8cd0fc-653b-4f07-a501-f121ec73b7d5-kube-api-access-k8ssv\") pod \"volume-data-source-validator-58c9f5f46d-zthx8\" (UID: \"cb8cd0fc-653b-4f07-a501-f121ec73b7d5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zthx8" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740737 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-run-netns\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740758 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740784 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6467adf-c26e-46ae-a212-98c9846bfe0e-ca-trust-extracted\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740825 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-trusted-ca\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740881 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct582\" (UniqueName: \"kubernetes.io/projected/9f173fd7-1ddb-4616-91b2-35a2bd6e0341-kube-api-access-ct582\") pod \"network-check-source-6f78d59f99-m2hhx\" (UID: \"9f173fd7-1ddb-4616-91b2-35a2bd6e0341\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m2hhx" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740874 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6467adf-c26e-46ae-a212-98c9846bfe0e-ca-trust-extracted\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740908 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhwr2\" (UniqueName: \"kubernetes.io/projected/d65ba521-c50d-47b4-82d9-1910dde4fbb1-kube-api-access-rhwr2\") pod \"node-resolver-5wq87\" (UID: \"d65ba521-c50d-47b4-82d9-1910dde4fbb1\") " pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740926 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740948 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740975 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbkft\" (UniqueName: \"kubernetes.io/projected/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-kube-api-access-jbkft\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.740999 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-image-registry-private-configuration\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741036 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-registration-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741064 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741110 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741130 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-kubernetes\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741147 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2-cert\") pod \"ingress-canary-x4wh2\" (UID: \"2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2\") " pod="openshift-ingress-canary/ingress-canary-x4wh2" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741158 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-trusted-ca\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741158 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741177 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-cni-bin\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741197 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-conf-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741187 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-image-registry-private-configuration\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741270 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phnxv\" (UniqueName: \"kubernetes.io/projected/9a35bad8-8af4-40c5-aa2b-1a939cc72c5e-kube-api-access-phnxv\") pod \"migrator-66df5d496b-tggjh\" (UID: \"9a35bad8-8af4-40c5-aa2b-1a939cc72c5e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tggjh" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741299 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741313 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9kvm\" (UniqueName: \"kubernetes.io/projected/b04c787c-4de0-483a-9f96-698fa046d04b-kube-api-access-q9kvm\") pod \"iptables-alerter-rc5dj\" (UID: \"b04c787c-4de0-483a-9f96-698fa046d04b\") " pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741345 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741373 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37a8184c-e9f5-4d96-a816-e0191c32ed06-serving-cert\") pod \"service-ca-operator-544d749c8c-r9qwk\" (UID: \"37a8184c-e9f5-4d96-a816-e0191c32ed06\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741385 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2-cert\") pod \"ingress-canary-x4wh2\" (UID: \"2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2\") " pod="openshift-ingress-canary/ingress-canary-x4wh2" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741493 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-systemd-units\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.741503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741522 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37a8184c-e9f5-4d96-a816-e0191c32ed06-serving-cert\") pod \"service-ca-operator-544d749c8c-r9qwk\" (UID: \"37a8184c-e9f5-4d96-a816-e0191c32ed06\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741531 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/58ec685b-55f0-42a2-9112-6fe8a813fd8a-dbus\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741554 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-system-cni-dir\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741588 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-installation-pull-secrets\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741604 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqf4v\" (UniqueName: \"kubernetes.io/projected/fb46e9cd-a477-4380-8d82-63a93d0f4099-kube-api-access-mqf4v\") pod \"node-ca-rwd6w\" (UID: \"fb46e9cd-a477-4380-8d82-63a93d0f4099\") " pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741622 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8fx4\" (UniqueName: \"kubernetes.io/projected/3d339ae6-a9e4-4258-8fe1-a393b943f953-kube-api-access-k8fx4\") pod \"service-ca-5fffc97bdf-rg8rf\" (UID: \"3d339ae6-a9e4-4258-8fe1-a393b943f953\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741769 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-sys\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741799 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741851 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-bound-sa-token\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741902 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37984c5f-7e58-4323-81f0-4bc7d485ab88-serving-cert\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741923 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovn-node-metrics-cert\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741925 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-installation-pull-secrets\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741941 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b04c787c-4de0-483a-9f96-698fa046d04b-host-slash\") pod \"iptables-alerter-rc5dj\" (UID: \"b04c787c-4de0-483a-9f96-698fa046d04b\") " pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.741991 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742005 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58ec685b-55f0-42a2-9112-6fe8a813fd8a-original-pull-secret\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742082 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f738a71-6d2b-4ade-ac92-990cf0a2b311-konnectivity-ca\") pod \"konnectivity-agent-vh4hd\" (UID: \"1f738a71-6d2b-4ade-ac92-990cf0a2b311\") " pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742124 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37984c5f-7e58-4323-81f0-4bc7d485ab88-tmp\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742175 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37984c5f-7e58-4323-81f0-4bc7d485ab88-tmp\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742212 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-node-log\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742236 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-netns\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742264 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6jjd\" (UniqueName: \"kubernetes.io/projected/add088ee-8843-4be8-ad89-f396481163c5-kube-api-access-w6jjd\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742294 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh5pc\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-kube-api-access-mh5pc\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742335 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-kubelet\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742376 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovn-node-metrics-cert\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742381 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37984c5f-7e58-4323-81f0-4bc7d485ab88-serving-cert\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742378 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f738a71-6d2b-4ade-ac92-990cf0a2b311-konnectivity-ca\") pod \"konnectivity-agent-vh4hd\" (UID: \"1f738a71-6d2b-4ade-ac92-990cf0a2b311\") " pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742405 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-cni-bin\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742440 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ktf5\" (UniqueName: \"kubernetes.io/projected/ba86e754-0fcc-46ab-a5e2-10609342eb13-kube-api-access-5ktf5\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742480 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-modprobe-d\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742501 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysconfig\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742546 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysctl-conf\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742574 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-systemd\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.742632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742610 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-default-certificate\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742700 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pwnr\" (UniqueName: \"kubernetes.io/projected/57dae9a4-2abc-4829-9beb-f61383f46bfe-kube-api-access-8pwnr\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742752 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-cni-multus\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742779 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-default-certificate\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742854 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmwp\" (UniqueName: \"kubernetes.io/projected/710eeea5-8bbf-4b10-8167-45a61512ef7f-kube-api-access-pwmwp\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742884 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-cni-netd\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742948 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/58ec685b-55f0-42a2-9112-6fe8a813fd8a-kubelet-config\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.742999 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743033 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjh7\" (UniqueName: \"kubernetes.io/projected/ddbb8fca-c02a-4e5f-a31f-919194f46f12-kube-api-access-fkjh7\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743063 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-config-volume\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743078 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb46e9cd-a477-4380-8d82-63a93d0f4099-serviceca\") pod \"node-ca-rwd6w\" (UID: \"fb46e9cd-a477-4380-8d82-63a93d0f4099\") " pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743098 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-socket-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743120 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab012cc6-ec86-41af-b6a1-e9278469c6de-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hplgb\" (UID: \"ab012cc6-ec86-41af-b6a1-e9278469c6de\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743140 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd1064c9-bade-4061-9b89-5d15e6a26a3d-ca-trust-extracted\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743162 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743191 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-stats-auth\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743220 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfcx5\" (UniqueName: \"kubernetes.io/projected/0852ff0f-d078-452f-bb56-3c13512f57ce-kube-api-access-xfcx5\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743227 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-config-volume\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743235 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd1064c9-bade-4061-9b89-5d15e6a26a3d-ca-trust-extracted\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743330 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-system-cni-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743359 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb46e9cd-a477-4380-8d82-63a93d0f4099-serviceca\") pod \"node-ca-rwd6w\" (UID: \"fb46e9cd-a477-4380-8d82-63a93d0f4099\") " pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743439 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-hostroot\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.743491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743469 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab012cc6-ec86-41af-b6a1-e9278469c6de-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hplgb\" (UID: \"ab012cc6-ec86-41af-b6a1-e9278469c6de\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743542 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3d339ae6-a9e4-4258-8fe1-a393b943f953-signing-cabundle\") pod \"service-ca-5fffc97bdf-rg8rf\" (UID: \"3d339ae6-a9e4-4258-8fe1-a393b943f953\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743616 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-stats-auth\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743625 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-log-socket\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743650 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovnkube-config\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743707 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs\") pod \"network-metrics-daemon-qvv5p\" (UID: \"2a406a0d-024a-411f-94c2-953a739c214f\") " pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743712 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3d339ae6-a9e4-4258-8fe1-a393b943f953-signing-cabundle\") pod \"service-ca-5fffc97bdf-rg8rf\" (UID: \"3d339ae6-a9e4-4258-8fe1-a393b943f953\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743813 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-host\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743846 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-ovnkube-config\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743845 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5cfd\" (UniqueName: \"kubernetes.io/projected/c5013527-41c2-4355-9204-594fe21d3c8a-kube-api-access-g5cfd\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743888 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2jlb\" (UniqueName: \"kubernetes.io/projected/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-api-access-n2jlb\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743915 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sldfp\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-kube-api-access-sldfp\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743938 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d65ba521-c50d-47b4-82d9-1910dde4fbb1-tmp-dir\") pod \"node-resolver-5wq87\" (UID: \"d65ba521-c50d-47b4-82d9-1910dde4fbb1\") " pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.743963 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3d339ae6-a9e4-4258-8fe1-a393b943f953-signing-key\") pod \"service-ca-5fffc97bdf-rg8rf\" (UID: \"3d339ae6-a9e4-4258-8fe1-a393b943f953\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744004 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f738a71-6d2b-4ade-ac92-990cf0a2b311-agent-certs\") pod \"konnectivity-agent-vh4hd\" (UID: \"1f738a71-6d2b-4ade-ac92-990cf0a2b311\") " pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744031 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-serving-cert\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744057 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-cnibin\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.744067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744066 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d65ba521-c50d-47b4-82d9-1910dde4fbb1-tmp-dir\") pod \"node-resolver-5wq87\" (UID: \"d65ba521-c50d-47b4-82d9-1910dde4fbb1\") " pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744083 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-data-volume\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744108 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744176 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3d339ae6-a9e4-4258-8fe1-a393b943f953-signing-key\") pod \"service-ca-5fffc97bdf-rg8rf\" (UID: \"3d339ae6-a9e4-4258-8fe1-a393b943f953\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744191 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9205351c-9ed0-4a10-8c52-0124c39a20fe-nginx-conf\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744211 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-serving-cert\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744219 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x526l\" (UniqueName: \"kubernetes.io/projected/1bc36693-102d-48e8-bb01-302b38055e05-kube-api-access-x526l\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744224 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-data-volume\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744238 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744269 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5013527-41c2-4355-9204-594fe21d3c8a-etc-tuned\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744289 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f738a71-6d2b-4ade-ac92-990cf0a2b311-agent-certs\") pod \"konnectivity-agent-vh4hd\" (UID: \"1f738a71-6d2b-4ade-ac92-990cf0a2b311\") " pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744297 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-image-registry-private-configuration\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744327 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744356 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5013527-41c2-4355-9204-594fe21d3c8a-etc-tuned\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744363 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9205351c-9ed0-4a10-8c52-0124c39a20fe-nginx-conf\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744357 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744382 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/add088ee-8843-4be8-ad89-f396481163c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744401 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5013527-41c2-4355-9204-594fe21d3c8a-tmp\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744443 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmt5p\" (UniqueName: \"kubernetes.io/projected/ab012cc6-ec86-41af-b6a1-e9278469c6de-kube-api-access-mmt5p\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hplgb\" (UID: \"ab012cc6-ec86-41af-b6a1-e9278469c6de\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744480 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-k8s-cni-cncf-io\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744489 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5013527-41c2-4355-9204-594fe21d3c8a-tmp\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744498 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-image-registry-private-configuration\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744508 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-certificates\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744555 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-installation-pull-secrets\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744595 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srsk\" (UniqueName: \"kubernetes.io/projected/43fb3f37-13b3-4eaa-9b92-6c343b1b2cef-kube-api-access-8srsk\") pod \"migrator-66df5d496b-h2s5x\" (UID: \"43fb3f37-13b3-4eaa-9b92-6c343b1b2cef\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h2s5x" Nov 03 21:29:45.744633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744637 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-certificates\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.745301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744707 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-device-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.745301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744735 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-sys-fs\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.745301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744766 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dl8\" (UniqueName: \"kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8\") pod \"network-check-target-pllqd\" (UID: \"ca003584-cd18-4740-a619-66c89853c773\") " pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:45.745301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744808 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-installation-pull-secrets\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.745301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744808 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9s65\" (UniqueName: \"kubernetes.io/projected/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2-kube-api-access-b9s65\") pod \"ingress-canary-x4wh2\" (UID: \"2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2\") " pod="openshift-ingress-canary/ingress-canary-x4wh2" Nov 03 21:29:45.745301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744850 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-var-lib-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.745301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744933 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-env-overrides\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.745301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.744998 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-certificates\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.745301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.745071 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d3ddfc6-6d13-4b06-983f-75cca78543ea-env-overrides\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.745301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.745116 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-certificates\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.758833 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.758809 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz5g7\" (UniqueName: \"kubernetes.io/projected/d8a307b9-8e43-4b73-941d-d5e2c035da14-kube-api-access-dz5g7\") pod \"kas-connection-checker-685895c7c9-7f6xj\" (UID: \"d8a307b9-8e43-4b73-941d-d5e2c035da14\") " pod="kube-system/kas-connection-checker-685895c7c9-7f6xj" Nov 03 21:29:45.758945 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.758929 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf9vr\" (UniqueName: \"kubernetes.io/projected/0d3ddfc6-6d13-4b06-983f-75cca78543ea-kube-api-access-cf9vr\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.759751 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.759734 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nc6h\" (UniqueName: \"kubernetes.io/projected/a734caac-db4c-4f6e-8dd0-415843ecd1b8-kube-api-access-7nc6h\") pod \"kas-connection-checker-685895c7c9-mq2cr\" (UID: \"a734caac-db4c-4f6e-8dd0-415843ecd1b8\") " pod="kube-system/kas-connection-checker-685895c7c9-mq2cr" Nov 03 21:29:45.760109 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.760091 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bqr\" (UniqueName: \"kubernetes.io/projected/2a406a0d-024a-411f-94c2-953a739c214f-kube-api-access-c8bqr\") pod \"network-metrics-daemon-qvv5p\" (UID: \"2a406a0d-024a-411f-94c2-953a739c214f\") " pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:45.760543 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.760523 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmdxx\" (UniqueName: \"kubernetes.io/projected/37984c5f-7e58-4323-81f0-4bc7d485ab88-kube-api-access-bmdxx\") pod \"insights-operator-7c4986d449-wxk7n\" (UID: \"37984c5f-7e58-4323-81f0-4bc7d485ab88\") " pod="openshift-insights/insights-operator-7c4986d449-wxk7n" Nov 03 21:29:45.760603 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.760559 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct582\" (UniqueName: \"kubernetes.io/projected/9f173fd7-1ddb-4616-91b2-35a2bd6e0341-kube-api-access-ct582\") pod \"network-check-source-6f78d59f99-m2hhx\" (UID: \"9f173fd7-1ddb-4616-91b2-35a2bd6e0341\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-m2hhx" Nov 03 21:29:45.760748 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.760733 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhwr2\" (UniqueName: \"kubernetes.io/projected/d65ba521-c50d-47b4-82d9-1910dde4fbb1-kube-api-access-rhwr2\") pod \"node-resolver-5wq87\" (UID: \"d65ba521-c50d-47b4-82d9-1910dde4fbb1\") " pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:45.762026 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.762008 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx68k\" (UniqueName: \"kubernetes.io/projected/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-kube-api-access-kx68k\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:45.762141 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.762124 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-bound-sa-token\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.762429 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.762414 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-bound-sa-token\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.762462 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.762448 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqf4v\" (UniqueName: \"kubernetes.io/projected/fb46e9cd-a477-4380-8d82-63a93d0f4099-kube-api-access-mqf4v\") pod \"node-ca-rwd6w\" (UID: \"fb46e9cd-a477-4380-8d82-63a93d0f4099\") " pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:45.763085 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.763067 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mddx\" (UniqueName: \"kubernetes.io/projected/37a8184c-e9f5-4d96-a816-e0191c32ed06-kube-api-access-6mddx\") pod \"service-ca-operator-544d749c8c-r9qwk\" (UID: \"37a8184c-e9f5-4d96-a816-e0191c32ed06\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" Nov 03 21:29:45.763533 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.763483 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phnxv\" (UniqueName: \"kubernetes.io/projected/9a35bad8-8af4-40c5-aa2b-1a939cc72c5e-kube-api-access-phnxv\") pod \"migrator-66df5d496b-tggjh\" (UID: \"9a35bad8-8af4-40c5-aa2b-1a939cc72c5e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tggjh" Nov 03 21:29:45.763614 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.763558 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ktf5\" (UniqueName: \"kubernetes.io/projected/ba86e754-0fcc-46ab-a5e2-10609342eb13-kube-api-access-5ktf5\") pod \"periodic-gathering-dflhb-2sf79\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:29:45.763652 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.763641 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9kvm\" (UniqueName: \"kubernetes.io/projected/b04c787c-4de0-483a-9f96-698fa046d04b-kube-api-access-q9kvm\") pod \"iptables-alerter-rc5dj\" (UID: \"b04c787c-4de0-483a-9f96-698fa046d04b\") " pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:45.763740 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.763701 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh5pc\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-kube-api-access-mh5pc\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.763740 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.763716 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbkft\" (UniqueName: \"kubernetes.io/projected/c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b-kube-api-access-jbkft\") pod \"console-operator-7cd56b8dbc-ltzz4\" (UID: \"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:45.764200 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.764179 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ssv\" (UniqueName: \"kubernetes.io/projected/cb8cd0fc-653b-4f07-a501-f121ec73b7d5-kube-api-access-k8ssv\") pod \"volume-data-source-validator-58c9f5f46d-zthx8\" (UID: \"cb8cd0fc-653b-4f07-a501-f121ec73b7d5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zthx8" Nov 03 21:29:45.764256 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.764240 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pwnr\" (UniqueName: \"kubernetes.io/projected/57dae9a4-2abc-4829-9beb-f61383f46bfe-kube-api-access-8pwnr\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:45.765597 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.764704 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btpnr\" (UniqueName: \"kubernetes.io/projected/efc25e42-1fb2-4fb4-b991-ce83b2d174d1-kube-api-access-btpnr\") pod \"kas-connection-checker-685895c7c9-pp685\" (UID: \"efc25e42-1fb2-4fb4-b991-ce83b2d174d1\") " pod="kube-system/kas-connection-checker-685895c7c9-pp685" Nov 03 21:29:45.765597 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.764958 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8fx4\" (UniqueName: \"kubernetes.io/projected/3d339ae6-a9e4-4258-8fe1-a393b943f953-kube-api-access-k8fx4\") pod \"service-ca-5fffc97bdf-rg8rf\" (UID: \"3d339ae6-a9e4-4258-8fe1-a393b943f953\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rg8rf" Nov 03 21:29:45.765597 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.765020 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x526l\" (UniqueName: \"kubernetes.io/projected/1bc36693-102d-48e8-bb01-302b38055e05-kube-api-access-x526l\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.765597 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.765113 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5cfd\" (UniqueName: \"kubernetes.io/projected/c5013527-41c2-4355-9204-594fe21d3c8a-kube-api-access-g5cfd\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.765597 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.765116 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sldfp\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-kube-api-access-sldfp\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.765597 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.765188 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwmwp\" (UniqueName: \"kubernetes.io/projected/710eeea5-8bbf-4b10-8167-45a61512ef7f-kube-api-access-pwmwp\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.765597 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.765197 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2jlb\" (UniqueName: \"kubernetes.io/projected/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-api-access-n2jlb\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:45.765597 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.765187 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmt5p\" (UniqueName: \"kubernetes.io/projected/ab012cc6-ec86-41af-b6a1-e9278469c6de-kube-api-access-mmt5p\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hplgb\" (UID: \"ab012cc6-ec86-41af-b6a1-e9278469c6de\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" Nov 03 21:29:45.765597 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.765196 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkjh7\" (UniqueName: \"kubernetes.io/projected/ddbb8fca-c02a-4e5f-a31f-919194f46f12-kube-api-access-fkjh7\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:45.765597 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.765543 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srsk\" (UniqueName: \"kubernetes.io/projected/43fb3f37-13b3-4eaa-9b92-6c343b1b2cef-kube-api-access-8srsk\") pod \"migrator-66df5d496b-h2s5x\" (UID: \"43fb3f37-13b3-4eaa-9b92-6c343b1b2cef\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h2s5x" Nov 03 21:29:45.765938 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.765620 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6jjd\" (UniqueName: \"kubernetes.io/projected/add088ee-8843-4be8-ad89-f396481163c5-kube-api-access-w6jjd\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.765938 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.765719 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfcx5\" (UniqueName: \"kubernetes.io/projected/0852ff0f-d078-452f-bb56-3c13512f57ce-kube-api-access-xfcx5\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:45.765938 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.765749 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7dl8\" (UniqueName: \"kubernetes.io/projected/ca003584-cd18-4740-a619-66c89853c773-kube-api-access-d7dl8\") pod \"network-check-target-pllqd\" (UID: \"ca003584-cd18-4740-a619-66c89853c773\") " pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:45.765938 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.765769 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9s65\" (UniqueName: \"kubernetes.io/projected/2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2-kube-api-access-b9s65\") pod \"ingress-canary-x4wh2\" (UID: \"2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2\") " pod="openshift-ingress-canary/ingress-canary-x4wh2" Nov 03 21:29:45.845786 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845727 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:45.845786 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845758 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-node-log\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.845786 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845773 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-netns\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.845786 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845789 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-kubelet\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845830 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-node-log\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845829 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-cni-bin\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845850 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-netns\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845874 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-cni-bin\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845859 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-modprobe-d\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845896 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysconfig\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845909 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysctl-conf\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845923 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-systemd\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.845933 5041 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.845949 5041 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd7fc569c-h759l: secret "image-registry-tls" not found Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845956 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-cni-multus\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845937 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-cni-multus\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845857 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-kubelet\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845947 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-modprobe-d\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845994 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-systemd\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845992 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-cni-netd\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.846008 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls podName:cd1064c9-bade-4061-9b89-5d15e6a26a3d nodeName:}" failed. No retries permitted until 2026-11-03 21:29:46.345988606 +0000 UTC m=+1.853216876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls") pod "image-registry-7fd7fc569c-h759l" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d") : secret "image-registry-tls" not found Nov 03 21:29:45.846030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.845993 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysconfig\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846037 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-cni-netd\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846048 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysctl-conf\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846086 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/58ec685b-55f0-42a2-9112-6fe8a813fd8a-kubelet-config\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846123 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.846182 5041 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846188 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/58ec685b-55f0-42a2-9112-6fe8a813fd8a-kubelet-config\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846232 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-socket-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.846255 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls podName:ddbb8fca-c02a-4e5f-a31f-919194f46f12 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:46.346243576 +0000 UTC m=+1.853471850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wq5rr" (UID: "ddbb8fca-c02a-4e5f-a31f-919194f46f12") : secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846283 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846285 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-socket-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846303 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-system-cni-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846328 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-hostroot\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846350 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-log-socket\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.846363 5041 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846375 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs\") pod \"network-metrics-daemon-qvv5p\" (UID: \"2a406a0d-024a-411f-94c2-953a739c214f\") " pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.846399 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls podName:2a8bade0-e7b8-4d61-b71f-fdb4b2517320 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:46.346386609 +0000 UTC m=+1.853614880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls") pod "dns-default-7t4b8" (UID: "2a8bade0-e7b8-4d61-b71f-fdb4b2517320") : secret "dns-default-metrics-tls" not found Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846401 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-hostroot\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846406 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-log-socket\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846415 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-host\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846425 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-system-cni-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846446 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-cnibin\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846472 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846487 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-host\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846505 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-cnibin\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846513 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846525 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846542 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846559 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-k8s-cni-cncf-io\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846575 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-device-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.846586 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle podName:0852ff0f-d078-452f-bb56-3c13512f57ce nodeName:}" failed. No retries permitted until 2026-11-03 21:29:46.346574092 +0000 UTC m=+1.853802380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle") pod "router-default-9f5c5b5d5-hpzcq" (UID: "0852ff0f-d078-452f-bb56-3c13512f57ce") : configmap references non-existent config key: service-ca.crt Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846598 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-k8s-cni-cncf-io\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.846618 5041 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846621 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-sys-fs\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846649 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-var-lib-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.846684 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls podName:57dae9a4-2abc-4829-9beb-f61383f46bfe nodeName:}" failed. No retries permitted until 2026-11-03 21:29:46.346654797 +0000 UTC m=+1.853883085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6fc7f" (UID: "57dae9a4-2abc-4829-9beb-f61383f46bfe") : secret "samples-operator-tls" not found Nov 03 21:29:45.846694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846710 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-var-lib-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846712 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-slash\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846736 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-slash\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846717 5041 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846752 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d65ba521-c50d-47b4-82d9-1910dde4fbb1-hosts-file\") pod \"node-resolver-5wq87\" (UID: \"d65ba521-c50d-47b4-82d9-1910dde4fbb1\") " pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846776 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-device-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846775 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846809 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-lib-modules\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846731 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-sys-fs\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846848 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d65ba521-c50d-47b4-82d9-1910dde4fbb1-hosts-file\") pod \"node-resolver-5wq87\" (UID: \"d65ba521-c50d-47b4-82d9-1910dde4fbb1\") " pod="openshift-dns/node-resolver-5wq87" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846848 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-etc-kubernetes\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846878 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-etc-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846817 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846880 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-etc-kubernetes\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846899 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-lib-modules\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846909 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846931 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-etc-openvswitch\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.846979 5041 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.846991 5041 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf564b6d-c6gsv: secret "image-registry-tls" not found Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847021 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb46e9cd-a477-4380-8d82-63a93d0f4099-host\") pod \"node-ca-rwd6w\" (UID: \"fb46e9cd-a477-4380-8d82-63a93d0f4099\") " pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.847025 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls podName:a6467adf-c26e-46ae-a212-98c9846bfe0e nodeName:}" failed. No retries permitted until 2026-11-03 21:29:46.347016687 +0000 UTC m=+1.854244945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls") pod "image-registry-7bf564b6d-c6gsv" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e") : secret "image-registry-tls" not found Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.846987 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb46e9cd-a477-4380-8d82-63a93d0f4099-host\") pod \"node-ca-rwd6w\" (UID: \"fb46e9cd-a477-4380-8d82-63a93d0f4099\") " pod="openshift-image-registry/node-ca-rwd6w" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847067 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-multus-certs\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847095 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-socket-dir-parent\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847138 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-socket-dir-parent\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847143 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-run-multus-certs\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847168 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-ovn\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847184 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847200 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-crio-socket\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847212 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-ovn\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847221 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-systemd\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847244 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-run-systemd\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847248 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-crio-socket\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847253 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-run\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847280 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-kubelet\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847305 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-run\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847307 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847334 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysctl-d\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847361 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847362 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-var-lib-kubelet\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847337 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-kubelet\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847389 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-cni-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847413 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-cnibin\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847424 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-var-lib-kubelet\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847440 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847465 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847474 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-cnibin\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847443 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-sysctl-d\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847492 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-run-netns\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847519 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847530 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-run-netns\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847555 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847485 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-cni-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.847559 5041 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847598 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847585 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-registration-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847579 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847623 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-registration-dir\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.847635 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls podName:7a8ffb5d-6f44-499a-b1bc-1cf127993dc7 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:46.347618702 +0000 UTC m=+1.854846972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-plrfk" (UID: "7a8ffb5d-6f44-499a-b1bc-1cf127993dc7") : secret "insights-runtime-extractor-tls" not found Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847654 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847696 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-kubernetes\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847711 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bc36693-102d-48e8-bb01-302b38055e05-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qdfv\" (UID: \"1bc36693-102d-48e8-bb01-302b38055e05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qdfv" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847722 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-cni-bin\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847747 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-conf-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847760 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-etc-kubernetes\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847774 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847778 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-host-var-lib-cni-bin\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847797 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/710eeea5-8bbf-4b10-8167-45a61512ef7f-multus-conf-dir\") pod \"multus-gdt8s\" (UID: \"710eeea5-8bbf-4b10-8167-45a61512ef7f\") " pod="openshift-multus/multus-gdt8s" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847799 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-systemd-units\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847825 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/58ec685b-55f0-42a2-9112-6fe8a813fd8a-dbus\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847850 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-system-cni-dir\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.847857 5041 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847870 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/58ec685b-55f0-42a2-9112-6fe8a813fd8a-dbus\") pod \"global-pull-secret-syncer-z5jjd\" (UID: \"58ec685b-55f0-42a2-9112-6fe8a813fd8a\") " pod="kube-system/global-pull-secret-syncer-z5jjd" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847875 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-sys\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847828 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d3ddfc6-6d13-4b06-983f-75cca78543ea-systemd-units\") pod \"ovnkube-node-ltgjc\" (UID: \"0d3ddfc6-6d13-4b06-983f-75cca78543ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:45.847904 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert podName:9205351c-9ed0-4a10-8c52-0124c39a20fe nodeName:}" failed. No retries permitted until 2026-11-03 21:29:46.347890317 +0000 UTC m=+1.855118587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w682j" (UID: "9205351c-9ed0-4a10-8c52-0124c39a20fe") : secret "networking-console-plugin-cert" not found Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847907 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/add088ee-8843-4be8-ad89-f396481163c5-system-cni-dir\") pod \"multus-additional-cni-plugins-2lwrg\" (UID: \"add088ee-8843-4be8-ad89-f396481163c5\") " pod="openshift-multus/multus-additional-cni-plugins-2lwrg" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847933 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b04c787c-4de0-483a-9f96-698fa046d04b-host-slash\") pod \"iptables-alerter-rc5dj\" (UID: \"b04c787c-4de0-483a-9f96-698fa046d04b\") " pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.847951 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5013527-41c2-4355-9204-594fe21d3c8a-sys\") pod \"tuned-cvc9c\" (UID: \"c5013527-41c2-4355-9204-594fe21d3c8a\") " pod="openshift-cluster-node-tuning-operator/tuned-cvc9c" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.848005 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b04c787c-4de0-483a-9f96-698fa046d04b-host-slash\") pod \"iptables-alerter-rc5dj\" (UID: \"b04c787c-4de0-483a-9f96-698fa046d04b\") " pod="openshift-network-operator/iptables-alerter-rc5dj" Nov 03 21:29:45.848106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.848065 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:45.850285 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.848894 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a406a0d-024a-411f-94c2-953a739c214f-metrics-certs\") pod \"network-metrics-daemon-qvv5p\" (UID: \"2a406a0d-024a-411f-94c2-953a739c214f\") " pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:45.850285 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.849613 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0852ff0f-d078-452f-bb56-3c13512f57ce-metrics-certs\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:45.914054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.914026 5041 scope.go:122] "RemoveContainer" containerID="5b4e56a0a125fb45430d89a491d2d78b22f384f0c1b89cd5dec41837a5111987" Nov 03 21:29:45.923740 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:45.923720 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvv5p" Nov 03 21:29:46.237037 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.237010 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x4wh2_2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2/serve-healthcheck-canary/0.log" Nov 03 21:29:46.254371 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.254343 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvv5p"] Nov 03 21:29:46.258545 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:46.258511 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a406a0d_024a_411f_94c2_953a739c214f.slice/crio-205af4ee00f6b26be710772075a1f931122510cb6b24d64f6b853e2e51f35815 WatchSource:0}: Error finding container 205af4ee00f6b26be710772075a1f931122510cb6b24d64f6b853e2e51f35815: Status 404 returned error can't find the container with id 205af4ee00f6b26be710772075a1f931122510cb6b24d64f6b853e2e51f35815 Nov 03 21:29:46.260483 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.260468 5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:29:46.352549 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.352519 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:46.352549 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.352552 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:46.352769 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.352582 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:46.352769 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.352611 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:46.352769 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:46.352698 5041 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:46.352769 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:46.352711 5041 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 21:29:46.352769 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:46.352730 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle podName:0852ff0f-d078-452f-bb56-3c13512f57ce nodeName:}" failed. No retries permitted until 2026-11-03 21:29:47.352710376 +0000 UTC m=+2.859938651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle") pod "router-default-9f5c5b5d5-hpzcq" (UID: "0852ff0f-d078-452f-bb56-3c13512f57ce") : configmap references non-existent config key: service-ca.crt Nov 03 21:29:46.352769 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:46.352715 5041 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf564b6d-c6gsv: secret "image-registry-tls" not found Nov 03 21:29:46.352769 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.352763 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:46.353095 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:46.352781 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls podName:a6467adf-c26e-46ae-a212-98c9846bfe0e nodeName:}" failed. No retries permitted until 2026-11-03 21:29:47.352766922 +0000 UTC m=+2.859995196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls") pod "image-registry-7bf564b6d-c6gsv" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e") : secret "image-registry-tls" not found Nov 03 21:29:46.353095 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:46.352800 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls podName:7a8ffb5d-6f44-499a-b1bc-1cf127993dc7 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:47.352791711 +0000 UTC m=+2.860019972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-plrfk" (UID: "7a8ffb5d-6f44-499a-b1bc-1cf127993dc7") : secret "insights-runtime-extractor-tls" not found Nov 03 21:29:46.353095 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:46.352834 5041 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 21:29:46.353095 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.352845 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:46.353095 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:46.352874 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert podName:9205351c-9ed0-4a10-8c52-0124c39a20fe nodeName:}" failed. No retries permitted until 2026-11-03 21:29:47.352861746 +0000 UTC m=+2.860090018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w682j" (UID: "9205351c-9ed0-4a10-8c52-0124c39a20fe") : secret "networking-console-plugin-cert" not found Nov 03 21:29:46.353095 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:46.352911 5041 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:46.353095 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:46.352921 5041 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd7fc569c-h759l: secret "image-registry-tls" not found Nov 03 21:29:46.353095 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.352922 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:46.353095 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:46.352952 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls podName:cd1064c9-bade-4061-9b89-5d15e6a26a3d nodeName:}" failed. No retries permitted until 2026-11-03 21:29:47.352943045 +0000 UTC m=+2.860171329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls") pod "image-registry-7fd7fc569c-h759l" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d") : secret "image-registry-tls" not found Nov 03 21:29:46.353095 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.352969 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:46.353095 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:46.352992 5041 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:46.353095 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:46.353026 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls podName:ddbb8fca-c02a-4e5f-a31f-919194f46f12 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:47.35301542 +0000 UTC m=+2.860243711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wq5rr" (UID: "ddbb8fca-c02a-4e5f-a31f-919194f46f12") : secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:46.353095 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:46.353067 5041 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 21:29:46.353485 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:46.353105 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls podName:2a8bade0-e7b8-4d61-b71f-fdb4b2517320 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:47.353094674 +0000 UTC m=+2.860322959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls") pod "dns-default-7t4b8" (UID: "2a8bade0-e7b8-4d61-b71f-fdb4b2517320") : secret "dns-default-metrics-tls" not found Nov 03 21:29:46.354852 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.354836 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/57dae9a4-2abc-4829-9beb-f61383f46bfe-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6fc7f\" (UID: \"57dae9a4-2abc-4829-9beb-f61383f46bfe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:46.436940 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.436919 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-h2s5x_43fb3f37-13b3-4eaa-9b92-6c343b1b2cef/migrator/0.log" Nov 03 21:29:46.514311 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.514250 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" Nov 03 21:29:46.637088 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.637043 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-tggjh_9a35bad8-8af4-40c5-aa2b-1a939cc72c5e/migrator/0.log" Nov 03 21:29:46.685637 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.685598 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvv5p" event={"ID":"2a406a0d-024a-411f-94c2-953a739c214f","Type":"ContainerStarted","Data":"205af4ee00f6b26be710772075a1f931122510cb6b24d64f6b853e2e51f35815"} Nov 03 21:29:46.687536 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.687508 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 21:29:46.688034 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.688012 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/1.log" Nov 03 21:29:46.688156 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.688054 5041 generic.go:356] "Generic (PLEG): container finished" podID="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" containerID="64f23fcbd6e769e674fe5b0d58926b6a88112534c1f3f4837ed3c4553e9fc617" exitCode=255 Nov 03 21:29:46.688352 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.688327 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" event={"ID":"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b","Type":"ContainerDied","Data":"64f23fcbd6e769e674fe5b0d58926b6a88112534c1f3f4837ed3c4553e9fc617"} Nov 03 21:29:46.688417 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.688394 5041 scope.go:122] "RemoveContainer" containerID="5b4e56a0a125fb45430d89a491d2d78b22f384f0c1b89cd5dec41837a5111987" Nov 03 21:29:46.688786 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.688759 5041 scope.go:122] "RemoveContainer" containerID="64f23fcbd6e769e674fe5b0d58926b6a88112534c1f3f4837ed3c4553e9fc617" Nov 03 21:29:46.688949 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:46.688930 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ltzz4_openshift-console-operator(c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" podUID="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" Nov 03 21:29:46.818469 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.817859 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f"] Nov 03 21:29:46.840806 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:46.840778 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-hplgb_ab012cc6-ec86-41af-b6a1-e9278469c6de/kube-storage-version-migrator-operator/0.log" Nov 03 21:29:47.364074 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:47.364041 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:47.364074 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:47.364084 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:47.364352 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:47.364109 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:47.364352 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:47.364132 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:47.364352 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:47.364176 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:47.364352 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:47.364207 5041 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:47.364352 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:47.364219 5041 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:47.364352 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:47.364242 5041 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd7fc569c-h759l: secret "image-registry-tls" not found Nov 03 21:29:47.364352 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:47.364271 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls podName:ddbb8fca-c02a-4e5f-a31f-919194f46f12 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:49.364255628 +0000 UTC m=+4.871483902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wq5rr" (UID: "ddbb8fca-c02a-4e5f-a31f-919194f46f12") : secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:47.364352 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:47.364274 5041 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:47.364352 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:47.364285 5041 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf564b6d-c6gsv: secret "image-registry-tls" not found Nov 03 21:29:47.364352 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:47.364290 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls podName:cd1064c9-bade-4061-9b89-5d15e6a26a3d nodeName:}" failed. No retries permitted until 2026-11-03 21:29:49.36427912 +0000 UTC m=+4.871507380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls") pod "image-registry-7fd7fc569c-h759l" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d") : secret "image-registry-tls" not found Nov 03 21:29:47.364352 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:47.364221 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:47.364352 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:47.364215 5041 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 21:29:47.364352 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:47.364319 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle podName:0852ff0f-d078-452f-bb56-3c13512f57ce nodeName:}" failed. No retries permitted until 2026-11-03 21:29:49.364299152 +0000 UTC m=+4.871527413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle") pod "router-default-9f5c5b5d5-hpzcq" (UID: "0852ff0f-d078-452f-bb56-3c13512f57ce") : configmap references non-existent config key: service-ca.crt Nov 03 21:29:47.364352 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:47.364328 5041 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 21:29:47.364877 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:47.364380 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls podName:a6467adf-c26e-46ae-a212-98c9846bfe0e nodeName:}" failed. No retries permitted until 2026-11-03 21:29:49.36436971 +0000 UTC m=+4.871597969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls") pod "image-registry-7bf564b6d-c6gsv" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e") : secret "image-registry-tls" not found Nov 03 21:29:47.364877 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:47.364399 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls podName:2a8bade0-e7b8-4d61-b71f-fdb4b2517320 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:49.364390562 +0000 UTC m=+4.871618844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls") pod "dns-default-7t4b8" (UID: "2a8bade0-e7b8-4d61-b71f-fdb4b2517320") : secret "dns-default-metrics-tls" not found Nov 03 21:29:47.364877 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:47.364425 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert podName:9205351c-9ed0-4a10-8c52-0124c39a20fe nodeName:}" failed. No retries permitted until 2026-11-03 21:29:49.364416992 +0000 UTC m=+4.871645257 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w682j" (UID: "9205351c-9ed0-4a10-8c52-0124c39a20fe") : secret "networking-console-plugin-cert" not found Nov 03 21:29:47.364877 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:47.364500 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:47.364877 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:47.364603 5041 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 21:29:47.364877 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:47.364641 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls podName:7a8ffb5d-6f44-499a-b1bc-1cf127993dc7 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:49.364633777 +0000 UTC m=+4.871862071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-plrfk" (UID: "7a8ffb5d-6f44-499a-b1bc-1cf127993dc7") : secret "insights-runtime-extractor-tls" not found Nov 03 21:29:47.693996 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:47.693964 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 21:29:47.696132 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:47.696104 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvv5p" event={"ID":"2a406a0d-024a-411f-94c2-953a739c214f","Type":"ContainerStarted","Data":"fad186c7efa9aa66b5961158fcbb20f25009324e1473ef40615cd61f9402999e"} Nov 03 21:29:47.696271 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:47.696169 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvv5p" event={"ID":"2a406a0d-024a-411f-94c2-953a739c214f","Type":"ContainerStarted","Data":"a3642e1103bc2bb482ba7725b4396b6f8bd10a10bf70c7f12c630c4d9af87cec"} Nov 03 21:29:47.697328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:47.697307 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" event={"ID":"57dae9a4-2abc-4829-9beb-f61383f46bfe","Type":"ContainerStarted","Data":"12b0d1249824eaa5f828e84d3de8576bff6bbcfa94a38fcf4d6167323a4efedd"} Nov 03 21:29:48.321458 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:48.321423 5041 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:48.321878 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:48.321852 5041 scope.go:122] "RemoveContainer" containerID="64f23fcbd6e769e674fe5b0d58926b6a88112534c1f3f4837ed3c4553e9fc617" Nov 03 21:29:48.322163 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:48.322132 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ltzz4_openshift-console-operator(c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" podUID="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" Nov 03 21:29:48.702712 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:48.702684 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" event={"ID":"57dae9a4-2abc-4829-9beb-f61383f46bfe","Type":"ContainerStarted","Data":"c6e790c10535fa6a5c2c4c4435e0fadd53715bfa6eb624b392dee0408a1a827d"} Nov 03 21:29:48.703102 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:48.702746 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6fc7f" event={"ID":"57dae9a4-2abc-4829-9beb-f61383f46bfe","Type":"ContainerStarted","Data":"1b59636692b1b810fc4831dfcb3f031aeac558990ab878e191495e62143cb2cf"} Nov 03 21:29:49.130226 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:49.130158 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:49.384570 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:49.384485 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:49.384570 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:49.384532 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:49.384570 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:49.384560 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:49.384828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:49.384601 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:49.384828 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:49.384636 5041 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:49.384828 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:49.384641 5041 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:49.384828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:49.384644 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:49.384828 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:49.384653 5041 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd7fc569c-h759l: secret "image-registry-tls" not found Nov 03 21:29:49.384828 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:49.384729 5041 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 21:29:49.384828 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:49.384738 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls podName:ddbb8fca-c02a-4e5f-a31f-919194f46f12 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:53.384718861 +0000 UTC m=+8.891947138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-wq5rr" (UID: "ddbb8fca-c02a-4e5f-a31f-919194f46f12") : secret "cluster-monitoring-operator-tls" not found Nov 03 21:29:49.384828 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:49.384775 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle podName:0852ff0f-d078-452f-bb56-3c13512f57ce nodeName:}" failed. No retries permitted until 2026-11-03 21:29:53.384764963 +0000 UTC m=+8.891993275 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle") pod "router-default-9f5c5b5d5-hpzcq" (UID: "0852ff0f-d078-452f-bb56-3c13512f57ce") : configmap references non-existent config key: service-ca.crt Nov 03 21:29:49.384828 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:49.384795 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls podName:cd1064c9-bade-4061-9b89-5d15e6a26a3d nodeName:}" failed. No retries permitted until 2026-11-03 21:29:53.38478495 +0000 UTC m=+8.892013208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls") pod "image-registry-7fd7fc569c-h759l" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d") : secret "image-registry-tls" not found Nov 03 21:29:49.384828 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:49.384814 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls podName:2a8bade0-e7b8-4d61-b71f-fdb4b2517320 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:53.384806218 +0000 UTC m=+8.892034484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls") pod "dns-default-7t4b8" (UID: "2a8bade0-e7b8-4d61-b71f-fdb4b2517320") : secret "dns-default-metrics-tls" not found Nov 03 21:29:49.384828 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:49.384815 5041 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 21:29:49.384828 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:49.384827 5041 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf564b6d-c6gsv: secret "image-registry-tls" not found Nov 03 21:29:49.385271 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:49.384860 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:49.385271 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:49.384894 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:49.385271 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:49.384902 5041 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 21:29:49.385271 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:49.384935 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls podName:a6467adf-c26e-46ae-a212-98c9846bfe0e nodeName:}" failed. No retries permitted until 2026-11-03 21:29:53.384921888 +0000 UTC m=+8.892150153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls") pod "image-registry-7bf564b6d-c6gsv" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e") : secret "image-registry-tls" not found Nov 03 21:29:49.385271 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:49.384953 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls podName:7a8ffb5d-6f44-499a-b1bc-1cf127993dc7 nodeName:}" failed. No retries permitted until 2026-11-03 21:29:53.384944408 +0000 UTC m=+8.892172686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-plrfk" (UID: "7a8ffb5d-6f44-499a-b1bc-1cf127993dc7") : secret "insights-runtime-extractor-tls" not found Nov 03 21:29:49.387237 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:49.387210 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9205351c-9ed0-4a10-8c52-0124c39a20fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w682j\" (UID: \"9205351c-9ed0-4a10-8c52-0124c39a20fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:49.515559 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:49.515529 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" Nov 03 21:29:49.707339 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:49.707309 5041 generic.go:356] "Generic (PLEG): container finished" podID="ab012cc6-ec86-41af-b6a1-e9278469c6de" containerID="565f24b8c71c179303ae03af26b133b4a5bb60d9b29a51996f08d35a2ef4de52" exitCode=0 Nov 03 21:29:49.707721 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:49.707383 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" event={"ID":"ab012cc6-ec86-41af-b6a1-e9278469c6de","Type":"ContainerDied","Data":"565f24b8c71c179303ae03af26b133b4a5bb60d9b29a51996f08d35a2ef4de52"} Nov 03 21:29:49.707829 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:49.707811 5041 scope.go:122] "RemoveContainer" containerID="565f24b8c71c179303ae03af26b133b4a5bb60d9b29a51996f08d35a2ef4de52" Nov 03 21:29:49.801945 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:49.801914 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-w682j"] Nov 03 21:29:49.805703 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:49.805632 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9205351c_9ed0_4a10_8c52_0124c39a20fe.slice/crio-096242df3e3f42c45123dee0f7b8c127ea0419b58f6d49ab7b6064b142a495a4 WatchSource:0}: Error finding container 096242df3e3f42c45123dee0f7b8c127ea0419b58f6d49ab7b6064b142a495a4: Status 404 returned error can't find the container with id 096242df3e3f42c45123dee0f7b8c127ea0419b58f6d49ab7b6064b142a495a4 Nov 03 21:29:50.505113 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:50.505079 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:50.505490 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:50.505475 5041 scope.go:122] "RemoveContainer" containerID="64f23fcbd6e769e674fe5b0d58926b6a88112534c1f3f4837ed3c4553e9fc617" Nov 03 21:29:50.505690 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:50.505655 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ltzz4_openshift-console-operator(c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" podUID="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" Nov 03 21:29:50.713075 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:50.713045 5041 generic.go:356] "Generic (PLEG): container finished" podID="37984c5f-7e58-4323-81f0-4bc7d485ab88" containerID="8cb75a53ae18d6a72660e08550ff834f5d5d0a0d7912eeb0ac4c8a859271eef9" exitCode=0 Nov 03 21:29:50.713521 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:50.713115 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wxk7n" event={"ID":"37984c5f-7e58-4323-81f0-4bc7d485ab88","Type":"ContainerDied","Data":"8cb75a53ae18d6a72660e08550ff834f5d5d0a0d7912eeb0ac4c8a859271eef9"} Nov 03 21:29:50.713521 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:50.713474 5041 scope.go:122] "RemoveContainer" containerID="8cb75a53ae18d6a72660e08550ff834f5d5d0a0d7912eeb0ac4c8a859271eef9" Nov 03 21:29:50.714596 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:50.714572 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" event={"ID":"9205351c-9ed0-4a10-8c52-0124c39a20fe","Type":"ContainerStarted","Data":"096242df3e3f42c45123dee0f7b8c127ea0419b58f6d49ab7b6064b142a495a4"} Nov 03 21:29:50.716381 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:50.716363 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hplgb" event={"ID":"ab012cc6-ec86-41af-b6a1-e9278469c6de","Type":"ContainerStarted","Data":"769f2bd8b9510e51fdd78ac0b07de562587701a475ea1c979e7565411cfb447f"} Nov 03 21:29:51.721750 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:51.721711 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wxk7n" event={"ID":"37984c5f-7e58-4323-81f0-4bc7d485ab88","Type":"ContainerStarted","Data":"6f5aab14ea5b7244d1257ad84a6a1b384410595a5b087a2f12841d7aaa3c9945"} Nov 03 21:29:51.723365 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:51.723338 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-w682j" event={"ID":"9205351c-9ed0-4a10-8c52-0124c39a20fe","Type":"ContainerStarted","Data":"05f1e828bbadee0db30be3f4be7d2280ef7ef6031e05aff743c06868cd2d0ff0"} Nov 03 21:29:52.737302 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:52.737271 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:29:53.422934 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.422893 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:53.423124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.422942 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:29:53.423124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.422972 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:53.423124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.423001 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:53.423124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.423030 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:53.423124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.423065 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:53.423394 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:53.423145 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle podName:0852ff0f-d078-452f-bb56-3c13512f57ce nodeName:}" failed. No retries permitted until 2026-11-03 21:30:01.423122823 +0000 UTC m=+16.930351100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle") pod "router-default-9f5c5b5d5-hpzcq" (UID: "0852ff0f-d078-452f-bb56-3c13512f57ce") : configmap references non-existent config key: service-ca.crt Nov 03 21:29:53.425460 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.425389 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls\") pod \"image-registry-7fd7fc569c-h759l\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:53.425460 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.425451 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a8bade0-e7b8-4d61-b71f-fdb4b2517320-metrics-tls\") pod \"dns-default-7t4b8\" (UID: \"2a8bade0-e7b8-4d61-b71f-fdb4b2517320\") " pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:53.425599 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.425451 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ffb5d-6f44-499a-b1bc-1cf127993dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-plrfk\" (UID: \"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7\") " pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:53.425837 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.425819 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls\") pod \"image-registry-7bf564b6d-c6gsv\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:53.425887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.425864 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ddbb8fca-c02a-4e5f-a31f-919194f46f12-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-wq5rr\" (UID: \"ddbb8fca-c02a-4e5f-a31f-919194f46f12\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:53.428738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.428719 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-plrfk" Nov 03 21:29:53.712946 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.712920 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" Nov 03 21:29:53.713138 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.712958 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:53.725513 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.725483 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:29:53.725766 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.725742 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:53.729730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:53.729709 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-plrfk"] Nov 03 21:29:54.019768 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:54.019694 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr"] Nov 03 21:29:54.022973 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:54.022944 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbb8fca_c02a_4e5f_a31f_919194f46f12.slice/crio-eeee0de699cc5c16dce6e4b3f05eab184be6dd360ce7c1f3c7e725233d48e4b2 WatchSource:0}: Error finding container eeee0de699cc5c16dce6e4b3f05eab184be6dd360ce7c1f3c7e725233d48e4b2: Status 404 returned error can't find the container with id eeee0de699cc5c16dce6e4b3f05eab184be6dd360ce7c1f3c7e725233d48e4b2 Nov 03 21:29:54.053437 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:54.053413 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7t4b8"] Nov 03 21:29:54.056801 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:54.056777 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8bade0_e7b8_4d61_b71f_fdb4b2517320.slice/crio-846aae7053cf48b9a7e730455b5c30c2a26b740f3ccc137a2b8336865e8fec37 WatchSource:0}: Error finding container 846aae7053cf48b9a7e730455b5c30c2a26b740f3ccc137a2b8336865e8fec37: Status 404 returned error can't find the container with id 846aae7053cf48b9a7e730455b5c30c2a26b740f3ccc137a2b8336865e8fec37 Nov 03 21:29:54.068171 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:54.068147 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fd7fc569c-h759l"] Nov 03 21:29:54.070721 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:54.070698 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1064c9_bade_4061_9b89_5d15e6a26a3d.slice/crio-e99618d9e4dd829d73a416ce55e282de272580d8dae224795ce6aec85a096e29 WatchSource:0}: Error finding container e99618d9e4dd829d73a416ce55e282de272580d8dae224795ce6aec85a096e29: Status 404 returned error can't find the container with id e99618d9e4dd829d73a416ce55e282de272580d8dae224795ce6aec85a096e29 Nov 03 21:29:54.073779 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:29:54.073733 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6467adf_c26e_46ae_a212_98c9846bfe0e.slice/crio-a2cbdef9537061f75df5f3e4a5f302627722eeb4999920ceabe193f9cca1397e WatchSource:0}: Error finding container a2cbdef9537061f75df5f3e4a5f302627722eeb4999920ceabe193f9cca1397e: Status 404 returned error can't find the container with id a2cbdef9537061f75df5f3e4a5f302627722eeb4999920ceabe193f9cca1397e Nov 03 21:29:54.081460 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:54.081407 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bf564b6d-c6gsv"] Nov 03 21:29:54.738697 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:54.738576 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7t4b8" event={"ID":"2a8bade0-e7b8-4d61-b71f-fdb4b2517320","Type":"ContainerStarted","Data":"846aae7053cf48b9a7e730455b5c30c2a26b740f3ccc137a2b8336865e8fec37"} Nov 03 21:29:54.741143 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:54.741046 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-plrfk" event={"ID":"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7","Type":"ContainerStarted","Data":"32cf95af1a14601e0db2757a35496c0f3ea0f9356d991887010402c8d3e1d0a1"} Nov 03 21:29:54.741143 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:54.741093 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-plrfk" event={"ID":"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7","Type":"ContainerStarted","Data":"f3770f660c70f379fd67d009c516324a00ff36577af0e02a187f4b85b218b43b"} Nov 03 21:29:54.741143 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:54.741112 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-plrfk" event={"ID":"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7","Type":"ContainerStarted","Data":"efd3941e8540fd456fe7d0d90c4bf422f3be69d2601fa7364dfeb5909b362da5"} Nov 03 21:29:54.742782 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:54.742756 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" event={"ID":"ddbb8fca-c02a-4e5f-a31f-919194f46f12","Type":"ContainerStarted","Data":"eeee0de699cc5c16dce6e4b3f05eab184be6dd360ce7c1f3c7e725233d48e4b2"} Nov 03 21:29:54.745011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:54.744940 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" event={"ID":"a6467adf-c26e-46ae-a212-98c9846bfe0e","Type":"ContainerStarted","Data":"c58ae1a7bbeb222f9ee0c711cccf377ee225617e4841defef85d38f67f6415ba"} Nov 03 21:29:54.745011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:54.744978 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" event={"ID":"a6467adf-c26e-46ae-a212-98c9846bfe0e","Type":"ContainerStarted","Data":"a2cbdef9537061f75df5f3e4a5f302627722eeb4999920ceabe193f9cca1397e"} Nov 03 21:29:54.747409 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:54.747390 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" event={"ID":"cd1064c9-bade-4061-9b89-5d15e6a26a3d","Type":"ContainerStarted","Data":"aef82c359ce080c35dd47109140cdd7653a4e720ef61d13af2dfc5f57eb39731"} Nov 03 21:29:54.747523 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:54.747429 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" event={"ID":"cd1064c9-bade-4061-9b89-5d15e6a26a3d","Type":"ContainerStarted","Data":"e99618d9e4dd829d73a416ce55e282de272580d8dae224795ce6aec85a096e29"} Nov 03 21:29:55.752566 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:55.751538 5041 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 21:29:55.752566 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:55.752148 5041 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 21:29:55.804009 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:55.803956 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:55.804204 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:55.804186 5041 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 21:29:55.810702 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:55.810680 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:29:56.409980 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:56.409913 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vh4hd" Nov 03 21:29:56.756611 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:56.756575 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7t4b8" event={"ID":"2a8bade0-e7b8-4d61-b71f-fdb4b2517320","Type":"ContainerStarted","Data":"1ea6bdbe0e097e521aed5a8944add4ab515bc351e03b2b4c79587353eea071d0"} Nov 03 21:29:56.757023 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:56.756641 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7t4b8" event={"ID":"2a8bade0-e7b8-4d61-b71f-fdb4b2517320","Type":"ContainerStarted","Data":"7d23aa0601490acb056fd4bc6b5e18504f3fe77375b3d68e77c2c749c450c4d0"} Nov 03 21:29:56.757023 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:56.756696 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7t4b8" Nov 03 21:29:56.758433 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:56.758408 5041 generic.go:356] "Generic (PLEG): container finished" podID="37984c5f-7e58-4323-81f0-4bc7d485ab88" containerID="6f5aab14ea5b7244d1257ad84a6a1b384410595a5b087a2f12841d7aaa3c9945" exitCode=0 Nov 03 21:29:56.758550 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:56.758473 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wxk7n" event={"ID":"37984c5f-7e58-4323-81f0-4bc7d485ab88","Type":"ContainerDied","Data":"6f5aab14ea5b7244d1257ad84a6a1b384410595a5b087a2f12841d7aaa3c9945"} Nov 03 21:29:56.758550 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:56.758523 5041 scope.go:122] "RemoveContainer" containerID="8cb75a53ae18d6a72660e08550ff834f5d5d0a0d7912eeb0ac4c8a859271eef9" Nov 03 21:29:56.758791 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:56.758774 5041 scope.go:122] "RemoveContainer" containerID="6f5aab14ea5b7244d1257ad84a6a1b384410595a5b087a2f12841d7aaa3c9945" Nov 03 21:29:56.758975 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:29:56.758950 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-wxk7n_openshift-insights(37984c5f-7e58-4323-81f0-4bc7d485ab88)\"" pod="openshift-insights/insights-operator-7c4986d449-wxk7n" podUID="37984c5f-7e58-4323-81f0-4bc7d485ab88" Nov 03 21:29:56.760549 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:56.760530 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-plrfk" event={"ID":"7a8ffb5d-6f44-499a-b1bc-1cf127993dc7","Type":"ContainerStarted","Data":"314d491f3446b923420e8441f1f3a02131dd1809331e466dd68bd3c103148883"} Nov 03 21:29:56.761903 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:56.761877 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-wq5rr" event={"ID":"ddbb8fca-c02a-4e5f-a31f-919194f46f12","Type":"ContainerStarted","Data":"beb0ff776b17e926e83aaee5f1c5637baad0ea625a1c3c9dd0d7b9737dd6ab2c"} Nov 03 21:29:57.605105 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:57.605079 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ltgjc" Nov 03 21:29:58.614522 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:58.614491 5041 scope.go:122] "RemoveContainer" containerID="64f23fcbd6e769e674fe5b0d58926b6a88112534c1f3f4837ed3c4553e9fc617" Nov 03 21:29:58.772610 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:58.772580 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 21:29:58.772796 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:58.772700 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" event={"ID":"c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b","Type":"ContainerStarted","Data":"870bf0be57334c44562ec2e785f508c8b33498369549a75ce2a05d3390d5f99c"} Nov 03 21:29:58.772995 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:58.772978 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:58.773797 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:58.773774 5041 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-ltzz4 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body= Nov 03 21:29:58.773875 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:58.773813 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" podUID="c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" Nov 03 21:29:59.778695 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.778646 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-pvgwb-whz8p"] Nov 03 21:29:59.782540 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.782513 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ltzz4" Nov 03 21:29:59.782733 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.782707 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pvgwb-whz8p" Nov 03 21:29:59.822637 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.822605 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-pvgwb-whz8p"] Nov 03 21:29:59.830142 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.830111 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fd7fc569c-h759l"] Nov 03 21:29:59.870432 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.870396 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjm5\" (UniqueName: \"kubernetes.io/projected/306f723b-4c8b-407b-80b9-26e1c1045c2f-kube-api-access-drjm5\") pod \"periodic-gathering-pvgwb-whz8p\" (UID: \"306f723b-4c8b-407b-80b9-26e1c1045c2f\") " pod="openshift-insights/periodic-gathering-pvgwb-whz8p" Nov 03 21:29:59.870594 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.870517 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/306f723b-4c8b-407b-80b9-26e1c1045c2f-archives-path\") pod \"periodic-gathering-pvgwb-whz8p\" (UID: \"306f723b-4c8b-407b-80b9-26e1c1045c2f\") " pod="openshift-insights/periodic-gathering-pvgwb-whz8p" Nov 03 21:29:59.870643 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.870627 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/306f723b-4c8b-407b-80b9-26e1c1045c2f-service-ca-bundle\") pod \"periodic-gathering-pvgwb-whz8p\" (UID: \"306f723b-4c8b-407b-80b9-26e1c1045c2f\") " pod="openshift-insights/periodic-gathering-pvgwb-whz8p" Nov 03 21:29:59.870712 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.870686 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/306f723b-4c8b-407b-80b9-26e1c1045c2f-trusted-ca-bundle\") pod \"periodic-gathering-pvgwb-whz8p\" (UID: \"306f723b-4c8b-407b-80b9-26e1c1045c2f\") " pod="openshift-insights/periodic-gathering-pvgwb-whz8p" Nov 03 21:29:59.971503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.971473 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/306f723b-4c8b-407b-80b9-26e1c1045c2f-trusted-ca-bundle\") pod \"periodic-gathering-pvgwb-whz8p\" (UID: \"306f723b-4c8b-407b-80b9-26e1c1045c2f\") " pod="openshift-insights/periodic-gathering-pvgwb-whz8p" Nov 03 21:29:59.971731 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.971523 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjm5\" (UniqueName: \"kubernetes.io/projected/306f723b-4c8b-407b-80b9-26e1c1045c2f-kube-api-access-drjm5\") pod \"periodic-gathering-pvgwb-whz8p\" (UID: \"306f723b-4c8b-407b-80b9-26e1c1045c2f\") " pod="openshift-insights/periodic-gathering-pvgwb-whz8p" Nov 03 21:29:59.971731 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.971641 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/306f723b-4c8b-407b-80b9-26e1c1045c2f-archives-path\") pod \"periodic-gathering-pvgwb-whz8p\" (UID: \"306f723b-4c8b-407b-80b9-26e1c1045c2f\") " pod="openshift-insights/periodic-gathering-pvgwb-whz8p" Nov 03 21:29:59.971948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.971755 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/306f723b-4c8b-407b-80b9-26e1c1045c2f-service-ca-bundle\") pod \"periodic-gathering-pvgwb-whz8p\" (UID: \"306f723b-4c8b-407b-80b9-26e1c1045c2f\") " pod="openshift-insights/periodic-gathering-pvgwb-whz8p" Nov 03 21:29:59.972222 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.972198 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/306f723b-4c8b-407b-80b9-26e1c1045c2f-archives-path\") pod \"periodic-gathering-pvgwb-whz8p\" (UID: \"306f723b-4c8b-407b-80b9-26e1c1045c2f\") " pod="openshift-insights/periodic-gathering-pvgwb-whz8p" Nov 03 21:29:59.972453 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.972437 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/306f723b-4c8b-407b-80b9-26e1c1045c2f-service-ca-bundle\") pod \"periodic-gathering-pvgwb-whz8p\" (UID: \"306f723b-4c8b-407b-80b9-26e1c1045c2f\") " pod="openshift-insights/periodic-gathering-pvgwb-whz8p" Nov 03 21:29:59.972840 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.972819 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/306f723b-4c8b-407b-80b9-26e1c1045c2f-trusted-ca-bundle\") pod \"periodic-gathering-pvgwb-whz8p\" (UID: \"306f723b-4c8b-407b-80b9-26e1c1045c2f\") " pod="openshift-insights/periodic-gathering-pvgwb-whz8p" Nov 03 21:29:59.991811 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:29:59.991790 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjm5\" (UniqueName: \"kubernetes.io/projected/306f723b-4c8b-407b-80b9-26e1c1045c2f-kube-api-access-drjm5\") pod \"periodic-gathering-pvgwb-whz8p\" (UID: \"306f723b-4c8b-407b-80b9-26e1c1045c2f\") " pod="openshift-insights/periodic-gathering-pvgwb-whz8p" Nov 03 21:30:00.094015 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:00.093942 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pvgwb-whz8p" Nov 03 21:30:00.392285 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:00.392208 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-pvgwb-whz8p"] Nov 03 21:30:00.395591 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:30:00.395565 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306f723b_4c8b_407b_80b9_26e1c1045c2f.slice/crio-30d24577972702e3fc61447417621c1c47cac989d78c0c46a4fb71521879cd5d WatchSource:0}: Error finding container 30d24577972702e3fc61447417621c1c47cac989d78c0c46a4fb71521879cd5d: Status 404 returned error can't find the container with id 30d24577972702e3fc61447417621c1c47cac989d78c0c46a4fb71521879cd5d Nov 03 21:30:00.783768 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:00.783729 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pvgwb-whz8p" event={"ID":"306f723b-4c8b-407b-80b9-26e1c1045c2f","Type":"ContainerStarted","Data":"ae16916df4e5810195066ef678b2217175adf87c335be835edc924233ea64792"} Nov 03 21:30:00.784247 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:00.783793 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pvgwb-whz8p" event={"ID":"306f723b-4c8b-407b-80b9-26e1c1045c2f","Type":"ContainerStarted","Data":"30d24577972702e3fc61447417621c1c47cac989d78c0c46a4fb71521879cd5d"} Nov 03 21:30:00.823992 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:00.823939 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-pvgwb-whz8p" podStartSLOduration=1.823926621 podStartE2EDuration="1.823926621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:00.823458191 +0000 UTC m=+16.330686472" watchObservedRunningTime="2026-11-03 21:30:00.823926621 +0000 UTC m=+16.331154900" Nov 03 21:30:01.484153 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:01.484112 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:30:01.484766 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:01.484745 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0852ff0f-d078-452f-bb56-3c13512f57ce-service-ca-bundle\") pod \"router-default-9f5c5b5d5-hpzcq\" (UID: \"0852ff0f-d078-452f-bb56-3c13512f57ce\") " pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:30:01.513121 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:01.513080 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:30:01.763294 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:01.763225 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:30:01.829908 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:01.829879 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-9f5c5b5d5-hpzcq"] Nov 03 21:30:01.833340 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:30:01.833312 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0852ff0f_d078_452f_bb56_3c13512f57ce.slice/crio-cdce77869d8880bc6de7fc8cfaab78a6efdbba682b0e7701d0437f6ac625b19f WatchSource:0}: Error finding container cdce77869d8880bc6de7fc8cfaab78a6efdbba682b0e7701d0437f6ac625b19f: Status 404 returned error can't find the container with id cdce77869d8880bc6de7fc8cfaab78a6efdbba682b0e7701d0437f6ac625b19f Nov 03 21:30:01.861602 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:01.861580 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7t4b8_2a8bade0-e7b8-4d61-b71f-fdb4b2517320/dns/0.log" Nov 03 21:30:02.061589 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:02.061518 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7t4b8_2a8bade0-e7b8-4d61-b71f-fdb4b2517320/kube-rbac-proxy/0.log" Nov 03 21:30:02.792870 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:02.792836 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" event={"ID":"0852ff0f-d078-452f-bb56-3c13512f57ce","Type":"ContainerStarted","Data":"60722365d6be55a350b0780269befebe7c0185ccdeb580aabe61b8f464d7f067"} Nov 03 21:30:02.793029 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:02.792892 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" event={"ID":"0852ff0f-d078-452f-bb56-3c13512f57ce","Type":"ContainerStarted","Data":"cdce77869d8880bc6de7fc8cfaab78a6efdbba682b0e7701d0437f6ac625b19f"} Nov 03 21:30:02.860931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:02.860903 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5wq87_d65ba521-c50d-47b4-82d9-1910dde4fbb1/dns-node-resolver/0.log" Nov 03 21:30:03.462253 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:03.462213 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7bf564b6d-c6gsv_a6467adf-c26e-46ae-a212-98c9846bfe0e/registry/0.log" Nov 03 21:30:03.513267 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:03.513236 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:30:03.515708 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:03.515685 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:30:03.661545 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:03.661513 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7fd7fc569c-h759l_cd1064c9-bade-4061-9b89-5d15e6a26a3d/registry/0.log" Nov 03 21:30:03.797320 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:03.797248 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:30:03.798418 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:03.798393 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-9f5c5b5d5-hpzcq" Nov 03 21:30:04.060720 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:04.060635 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rwd6w_fb46e9cd-a477-4380-8d82-63a93d0f4099/node-ca/0.log" Nov 03 21:30:04.461508 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:04.461482 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9f5c5b5d5-hpzcq_0852ff0f-d078-452f-bb56-3c13512f57ce/router/0.log" Nov 03 21:30:05.060884 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:05.060842 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x4wh2_2d4f0a4f-af3b-4bdc-89f1-b8716c76ded2/serve-healthcheck-canary/0.log" Nov 03 21:30:06.769229 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:06.769200 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7t4b8" Nov 03 21:30:07.758548 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.758517 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xs5ht"] Nov 03 21:30:07.769697 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.769673 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.809647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.809615 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 21:30:07.809647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.809616 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 21:30:07.809907 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.809890 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lp7fr\"" Nov 03 21:30:07.823191 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.823165 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 21:30:07.823473 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.823457 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 21:30:07.831158 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.831134 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afd2b194-1d73-415c-913d-4d998a51ea4b-node-exporter-textfile\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.831316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.831162 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afd2b194-1d73-415c-913d-4d998a51ea4b-node-exporter-tls\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.831316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.831183 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afd2b194-1d73-415c-913d-4d998a51ea4b-sys\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.831316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.831261 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afd2b194-1d73-415c-913d-4d998a51ea4b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.831464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.831353 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afd2b194-1d73-415c-913d-4d998a51ea4b-root\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.831464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.831405 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afd2b194-1d73-415c-913d-4d998a51ea4b-node-exporter-wtmp\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.831464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.831427 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afd2b194-1d73-415c-913d-4d998a51ea4b-metrics-client-ca\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.831464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.831444 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fprcw\" (UniqueName: \"kubernetes.io/projected/afd2b194-1d73-415c-913d-4d998a51ea4b-kube-api-access-fprcw\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.831650 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.831527 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afd2b194-1d73-415c-913d-4d998a51ea4b-node-exporter-accelerators-collector-config\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.932322 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.932288 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afd2b194-1d73-415c-913d-4d998a51ea4b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.932488 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.932336 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afd2b194-1d73-415c-913d-4d998a51ea4b-root\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.932488 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.932374 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afd2b194-1d73-415c-913d-4d998a51ea4b-node-exporter-wtmp\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.932488 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.932397 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afd2b194-1d73-415c-913d-4d998a51ea4b-metrics-client-ca\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.932488 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.932424 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fprcw\" (UniqueName: \"kubernetes.io/projected/afd2b194-1d73-415c-913d-4d998a51ea4b-kube-api-access-fprcw\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.932488 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.932459 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afd2b194-1d73-415c-913d-4d998a51ea4b-node-exporter-accelerators-collector-config\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.932488 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.932465 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afd2b194-1d73-415c-913d-4d998a51ea4b-root\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.932802 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.932502 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afd2b194-1d73-415c-913d-4d998a51ea4b-node-exporter-textfile\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.932802 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.932538 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afd2b194-1d73-415c-913d-4d998a51ea4b-node-exporter-tls\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.932802 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.932575 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afd2b194-1d73-415c-913d-4d998a51ea4b-sys\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.932802 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.932574 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afd2b194-1d73-415c-913d-4d998a51ea4b-node-exporter-wtmp\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.932802 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.932710 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afd2b194-1d73-415c-913d-4d998a51ea4b-sys\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.933053 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.933024 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afd2b194-1d73-415c-913d-4d998a51ea4b-node-exporter-textfile\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.933209 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.933184 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afd2b194-1d73-415c-913d-4d998a51ea4b-metrics-client-ca\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.933250 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.933204 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afd2b194-1d73-415c-913d-4d998a51ea4b-node-exporter-accelerators-collector-config\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.934830 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.934805 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afd2b194-1d73-415c-913d-4d998a51ea4b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.934930 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.934844 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afd2b194-1d73-415c-913d-4d998a51ea4b-node-exporter-tls\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:07.953200 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:07.953178 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fprcw\" (UniqueName: \"kubernetes.io/projected/afd2b194-1d73-415c-913d-4d998a51ea4b-kube-api-access-fprcw\") pod \"node-exporter-xs5ht\" (UID: \"afd2b194-1d73-415c-913d-4d998a51ea4b\") " pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:08.090205 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:08.090141 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xs5ht" Nov 03 21:30:08.121615 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:30:08.121587 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd2b194_1d73_415c_913d_4d998a51ea4b.slice/crio-41c650ac847cf24bdaa45804a410dc4682ee999c5cc92fe01dc2ef10aa4a257d WatchSource:0}: Error finding container 41c650ac847cf24bdaa45804a410dc4682ee999c5cc92fe01dc2ef10aa4a257d: Status 404 returned error can't find the container with id 41c650ac847cf24bdaa45804a410dc4682ee999c5cc92fe01dc2ef10aa4a257d Nov 03 21:30:08.819006 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:08.818974 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xs5ht" event={"ID":"afd2b194-1d73-415c-913d-4d998a51ea4b","Type":"ContainerStarted","Data":"41c650ac847cf24bdaa45804a410dc4682ee999c5cc92fe01dc2ef10aa4a257d"} Nov 03 21:30:09.613910 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:09.613875 5041 scope.go:122] "RemoveContainer" containerID="6f5aab14ea5b7244d1257ad84a6a1b384410595a5b087a2f12841d7aaa3c9945" Nov 03 21:30:09.825324 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:09.825283 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wxk7n" event={"ID":"37984c5f-7e58-4323-81f0-4bc7d485ab88","Type":"ContainerStarted","Data":"4662d90efb3be3f61d92d1ab5d269eb729e98b6cecfed3e0e6c867b3b780b0b5"} Nov 03 21:30:09.826935 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:09.826902 5041 generic.go:356] "Generic (PLEG): container finished" podID="afd2b194-1d73-415c-913d-4d998a51ea4b" containerID="440e6e3f66b66183bfbee98ba0815e8b0fdaab303ac8fe51737ad52161bbfa83" exitCode=0 Nov 03 21:30:09.827063 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:09.826940 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xs5ht" event={"ID":"afd2b194-1d73-415c-913d-4d998a51ea4b","Type":"ContainerDied","Data":"440e6e3f66b66183bfbee98ba0815e8b0fdaab303ac8fe51737ad52161bbfa83"} Nov 03 21:30:09.835848 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:09.835815 5041 patch_prober.go:29] interesting pod/image-registry-7fd7fc569c-h759l container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 21:30:09.835938 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:09.835865 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" podUID="cd1064c9-bade-4061-9b89-5d15e6a26a3d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:30:10.180361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:10.180329 5041 patch_prober.go:29] interesting pod/image-registry-7bf564b6d-c6gsv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 21:30:10.180522 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:10.180379 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" podUID="a6467adf-c26e-46ae-a212-98c9846bfe0e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:30:10.775714 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:30:10.775681 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:30:10.833119 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:10.833089 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xs5ht" event={"ID":"afd2b194-1d73-415c-913d-4d998a51ea4b","Type":"ContainerStarted","Data":"8b95e02a242ab2eec3d840e20f45fed0313913a0717add370f7563f1c16cce4f"} Nov 03 21:30:10.833449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:10.833137 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xs5ht" event={"ID":"afd2b194-1d73-415c-913d-4d998a51ea4b","Type":"ContainerStarted","Data":"cef259fd16a75c4e856b56cf39b131b0fed90e1829576bcf54c6c29309f724b6"} Nov 03 21:30:10.874361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:10.874317 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xs5ht" podStartSLOduration=2.469786663 podStartE2EDuration="3.874304993s" totalImagesPullingTime="1.314891974s" totalInitContainerRuntime="89.626356ms" isStatefulPod=true podCreationTimestamp="2026-11-03 21:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:10.874196796 +0000 UTC m=+26.381425072" watchObservedRunningTime="2026-11-03 21:30:10.874304993 +0000 UTC m=+26.381533271" Nov 03 21:30:11.767699 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:11.767649 5041 patch_prober.go:29] interesting pod/image-registry-7bf564b6d-c6gsv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 21:30:11.767863 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:11.767713 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" podUID="a6467adf-c26e-46ae-a212-98c9846bfe0e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:30:12.740246 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:12.740217 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pllqd" Nov 03 21:30:16.559027 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:16.558997 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bf564b6d-c6gsv"] Nov 03 21:30:16.563274 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:16.563249 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:30:19.834279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:19.834249 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:30:24.850708 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:24.850633 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" podUID="cd1064c9-bade-4061-9b89-5d15e6a26a3d" containerName="registry" containerID="cri-o://aef82c359ce080c35dd47109140cdd7653a4e720ef61d13af2dfc5f57eb39731" gracePeriod=30 Nov 03 21:30:25.096381 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.096358 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:30:25.182261 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.182169 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1064c9-bade-4061-9b89-5d15e6a26a3d-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd1064c9-bade-4061-9b89-5d15e6a26a3d-ca-trust-extracted\") pod \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " Nov 03 21:30:25.182261 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.182235 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-certificates\") pod \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " Nov 03 21:30:25.182470 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.182306 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-installation-pull-secrets\") pod \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " Nov 03 21:30:25.182470 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.182350 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-bound-sa-token\") pod \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " Nov 03 21:30:25.182470 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.182380 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-image-registry-private-configuration\") pod \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " Nov 03 21:30:25.182470 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.182414 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-trusted-ca\") pod \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " Nov 03 21:30:25.182470 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.182443 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls\") pod \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " Nov 03 21:30:25.182805 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.182501 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-kube-api-access-mh5pc\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-kube-api-access-mh5pc\") pod \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\" (UID: \"cd1064c9-bade-4061-9b89-5d15e6a26a3d\") " Nov 03 21:30:25.182805 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.182626 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-certificates" pod "cd1064c9-bade-4061-9b89-5d15e6a26a3d" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:25.182991 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.182947 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-trusted-ca" pod "cd1064c9-bade-4061-9b89-5d15e6a26a3d" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:25.183191 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.183135 5041 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-certificates\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:25.185104 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.185071 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-kube-api-access-mh5pc" pod "cd1064c9-bade-4061-9b89-5d15e6a26a3d" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d"). InnerVolumeSpecName "kube-api-access-mh5pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:25.185104 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.185092 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-installation-pull-secrets" pod "cd1064c9-bade-4061-9b89-5d15e6a26a3d" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:30:25.185263 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.185159 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-bound-sa-token" pod "cd1064c9-bade-4061-9b89-5d15e6a26a3d" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:25.185263 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.185178 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls" pod "cd1064c9-bade-4061-9b89-5d15e6a26a3d" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:25.185263 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.185254 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-image-registry-private-configuration" pod "cd1064c9-bade-4061-9b89-5d15e6a26a3d" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:30:25.189222 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.189202 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1064c9-bade-4061-9b89-5d15e6a26a3d-ca-trust-extracted" pod "cd1064c9-bade-4061-9b89-5d15e6a26a3d" (UID: "cd1064c9-bade-4061-9b89-5d15e6a26a3d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:30:25.283759 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.283724 5041 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-bound-sa-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:25.283759 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.283752 5041 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-image-registry-private-configuration\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:25.283759 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.283763 5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd1064c9-bade-4061-9b89-5d15e6a26a3d-trusted-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:25.283963 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.283772 5041 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-registry-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:25.283963 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.283780 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh5pc\" (UniqueName: \"kubernetes.io/projected/cd1064c9-bade-4061-9b89-5d15e6a26a3d-kube-api-access-mh5pc\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:25.283963 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.283789 5041 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd1064c9-bade-4061-9b89-5d15e6a26a3d-ca-trust-extracted\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:25.283963 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.283798 5041 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd1064c9-bade-4061-9b89-5d15e6a26a3d-installation-pull-secrets\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:25.893085 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.893001 5041 generic.go:356] "Generic (PLEG): container finished" podID="cd1064c9-bade-4061-9b89-5d15e6a26a3d" containerID="aef82c359ce080c35dd47109140cdd7653a4e720ef61d13af2dfc5f57eb39731" exitCode=0 Nov 03 21:30:25.893085 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.893050 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" Nov 03 21:30:25.893085 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.893079 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" event={"ID":"cd1064c9-bade-4061-9b89-5d15e6a26a3d","Type":"ContainerDied","Data":"aef82c359ce080c35dd47109140cdd7653a4e720ef61d13af2dfc5f57eb39731"} Nov 03 21:30:25.893616 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.893119 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd7fc569c-h759l" event={"ID":"cd1064c9-bade-4061-9b89-5d15e6a26a3d","Type":"ContainerDied","Data":"e99618d9e4dd829d73a416ce55e282de272580d8dae224795ce6aec85a096e29"} Nov 03 21:30:25.893616 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.893139 5041 scope.go:122] "RemoveContainer" containerID="aef82c359ce080c35dd47109140cdd7653a4e720ef61d13af2dfc5f57eb39731" Nov 03 21:30:25.902229 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.902211 5041 scope.go:122] "RemoveContainer" containerID="aef82c359ce080c35dd47109140cdd7653a4e720ef61d13af2dfc5f57eb39731" Nov 03 21:30:25.902510 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:30:25.902487 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef82c359ce080c35dd47109140cdd7653a4e720ef61d13af2dfc5f57eb39731\": container with ID starting with aef82c359ce080c35dd47109140cdd7653a4e720ef61d13af2dfc5f57eb39731 not found: ID does not exist" containerID="aef82c359ce080c35dd47109140cdd7653a4e720ef61d13af2dfc5f57eb39731" Nov 03 21:30:25.902556 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.902520 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef82c359ce080c35dd47109140cdd7653a4e720ef61d13af2dfc5f57eb39731"} err="failed to get container status \"aef82c359ce080c35dd47109140cdd7653a4e720ef61d13af2dfc5f57eb39731\": rpc error: code = NotFound desc = could not find container \"aef82c359ce080c35dd47109140cdd7653a4e720ef61d13af2dfc5f57eb39731\": container with ID starting with aef82c359ce080c35dd47109140cdd7653a4e720ef61d13af2dfc5f57eb39731 not found: ID does not exist" Nov 03 21:30:25.960887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.960852 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fd7fc569c-h759l"] Nov 03 21:30:25.964642 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:25.964625 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7fd7fc569c-h759l"] Nov 03 21:30:26.618805 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:26.618772 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd1064c9-bade-4061-9b89-5d15e6a26a3d" path="/var/lib/kubelet/pods/cd1064c9-bade-4061-9b89-5d15e6a26a3d/volumes" Nov 03 21:30:32.544230 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.544191 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-pvgwb-whz8p"] Nov 03 21:30:32.544843 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.544441 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-pvgwb-whz8p" podUID="306f723b-4c8b-407b-80b9-26e1c1045c2f" containerName="insights-gathering" containerID="cri-o://ae16916df4e5810195066ef678b2217175adf87c335be835edc924233ea64792" gracePeriod=30 Nov 03 21:30:32.547814 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.547790 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-dflhb-2sf79"] Nov 03 21:30:32.547997 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.547976 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-dflhb-2sf79" podUID="ba86e754-0fcc-46ab-a5e2-10609342eb13" containerName="insights-gathering" containerID="cri-o://e16de07c8a8d6c890d059e74651fd941023879c54134540c9620a847e99d1dfd" gracePeriod=30 Nov 03 21:30:32.774151 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.774120 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pvgwb-whz8p" Nov 03 21:30:32.805169 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.805149 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:30:32.843250 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.843227 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-service-ca-bundle\") pod \"ba86e754-0fcc-46ab-a5e2-10609342eb13\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " Nov 03 21:30:32.843376 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.843269 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306f723b-4c8b-407b-80b9-26e1c1045c2f-kube-api-access-drjm5\" (UniqueName: \"kubernetes.io/projected/306f723b-4c8b-407b-80b9-26e1c1045c2f-kube-api-access-drjm5\") pod \"306f723b-4c8b-407b-80b9-26e1c1045c2f\" (UID: \"306f723b-4c8b-407b-80b9-26e1c1045c2f\") " Nov 03 21:30:32.843376 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.843318 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/306f723b-4c8b-407b-80b9-26e1c1045c2f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/306f723b-4c8b-407b-80b9-26e1c1045c2f-service-ca-bundle\") pod \"306f723b-4c8b-407b-80b9-26e1c1045c2f\" (UID: \"306f723b-4c8b-407b-80b9-26e1c1045c2f\") " Nov 03 21:30:32.843376 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.843338 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba86e754-0fcc-46ab-a5e2-10609342eb13-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ba86e754-0fcc-46ab-a5e2-10609342eb13-archives-path\") pod \"ba86e754-0fcc-46ab-a5e2-10609342eb13\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " Nov 03 21:30:32.843376 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.843361 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306f723b-4c8b-407b-80b9-26e1c1045c2f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/306f723b-4c8b-407b-80b9-26e1c1045c2f-archives-path\") pod \"306f723b-4c8b-407b-80b9-26e1c1045c2f\" (UID: \"306f723b-4c8b-407b-80b9-26e1c1045c2f\") " Nov 03 21:30:32.843575 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.843403 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba86e754-0fcc-46ab-a5e2-10609342eb13-kube-api-access-5ktf5\" (UniqueName: \"kubernetes.io/projected/ba86e754-0fcc-46ab-a5e2-10609342eb13-kube-api-access-5ktf5\") pod \"ba86e754-0fcc-46ab-a5e2-10609342eb13\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " Nov 03 21:30:32.843575 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.843437 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/306f723b-4c8b-407b-80b9-26e1c1045c2f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/306f723b-4c8b-407b-80b9-26e1c1045c2f-trusted-ca-bundle\") pod \"306f723b-4c8b-407b-80b9-26e1c1045c2f\" (UID: \"306f723b-4c8b-407b-80b9-26e1c1045c2f\") " Nov 03 21:30:32.843575 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.843506 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-trusted-ca-bundle\") pod \"ba86e754-0fcc-46ab-a5e2-10609342eb13\" (UID: \"ba86e754-0fcc-46ab-a5e2-10609342eb13\") " Nov 03 21:30:32.843746 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.843631 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-service-ca-bundle" pod "ba86e754-0fcc-46ab-a5e2-10609342eb13" (UID: "ba86e754-0fcc-46ab-a5e2-10609342eb13"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:32.843746 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.843712 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba86e754-0fcc-46ab-a5e2-10609342eb13-archives-path" pod "ba86e754-0fcc-46ab-a5e2-10609342eb13" (UID: "ba86e754-0fcc-46ab-a5e2-10609342eb13"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:30:32.843746 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.843734 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306f723b-4c8b-407b-80b9-26e1c1045c2f-archives-path" pod "306f723b-4c8b-407b-80b9-26e1c1045c2f" (UID: "306f723b-4c8b-407b-80b9-26e1c1045c2f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:30:32.843895 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.843799 5041 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-service-ca-bundle\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:32.843895 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.843817 5041 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ba86e754-0fcc-46ab-a5e2-10609342eb13-archives-path\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:32.843998 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.843904 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-trusted-ca-bundle" pod "ba86e754-0fcc-46ab-a5e2-10609342eb13" (UID: "ba86e754-0fcc-46ab-a5e2-10609342eb13"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:32.843998 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.843948 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306f723b-4c8b-407b-80b9-26e1c1045c2f-trusted-ca-bundle" pod "306f723b-4c8b-407b-80b9-26e1c1045c2f" (UID: "306f723b-4c8b-407b-80b9-26e1c1045c2f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:32.844123 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.844103 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306f723b-4c8b-407b-80b9-26e1c1045c2f-service-ca-bundle" pod "306f723b-4c8b-407b-80b9-26e1c1045c2f" (UID: "306f723b-4c8b-407b-80b9-26e1c1045c2f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:32.846180 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.846161 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306f723b-4c8b-407b-80b9-26e1c1045c2f-kube-api-access-drjm5" pod "306f723b-4c8b-407b-80b9-26e1c1045c2f" (UID: "306f723b-4c8b-407b-80b9-26e1c1045c2f"). InnerVolumeSpecName "kube-api-access-drjm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:32.846263 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.846246 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba86e754-0fcc-46ab-a5e2-10609342eb13-kube-api-access-5ktf5" pod "ba86e754-0fcc-46ab-a5e2-10609342eb13" (UID: "ba86e754-0fcc-46ab-a5e2-10609342eb13"). InnerVolumeSpecName "kube-api-access-5ktf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:32.921841 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.921810 5041 generic.go:356] "Generic (PLEG): container finished" podID="306f723b-4c8b-407b-80b9-26e1c1045c2f" containerID="ae16916df4e5810195066ef678b2217175adf87c335be835edc924233ea64792" exitCode=2 Nov 03 21:30:32.921981 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.921859 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pvgwb-whz8p" Nov 03 21:30:32.921981 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.921896 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pvgwb-whz8p" event={"ID":"306f723b-4c8b-407b-80b9-26e1c1045c2f","Type":"ContainerDied","Data":"ae16916df4e5810195066ef678b2217175adf87c335be835edc924233ea64792"} Nov 03 21:30:32.921981 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.921938 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pvgwb-whz8p" event={"ID":"306f723b-4c8b-407b-80b9-26e1c1045c2f","Type":"ContainerDied","Data":"30d24577972702e3fc61447417621c1c47cac989d78c0c46a4fb71521879cd5d"} Nov 03 21:30:32.921981 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.921963 5041 scope.go:122] "RemoveContainer" containerID="ae16916df4e5810195066ef678b2217175adf87c335be835edc924233ea64792" Nov 03 21:30:32.923265 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.923244 5041 generic.go:356] "Generic (PLEG): container finished" podID="ba86e754-0fcc-46ab-a5e2-10609342eb13" containerID="e16de07c8a8d6c890d059e74651fd941023879c54134540c9620a847e99d1dfd" exitCode=2 Nov 03 21:30:32.923397 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.923291 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dflhb-2sf79" Nov 03 21:30:32.923397 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.923314 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dflhb-2sf79" event={"ID":"ba86e754-0fcc-46ab-a5e2-10609342eb13","Type":"ContainerDied","Data":"e16de07c8a8d6c890d059e74651fd941023879c54134540c9620a847e99d1dfd"} Nov 03 21:30:32.923397 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.923349 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dflhb-2sf79" event={"ID":"ba86e754-0fcc-46ab-a5e2-10609342eb13","Type":"ContainerDied","Data":"bcf1d521ab66e7d26743f70d0164ebeb81aef32e0cea06893d82ceb83b8ce414"} Nov 03 21:30:32.932233 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.932212 5041 scope.go:122] "RemoveContainer" containerID="ae16916df4e5810195066ef678b2217175adf87c335be835edc924233ea64792" Nov 03 21:30:32.932588 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:30:32.932562 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae16916df4e5810195066ef678b2217175adf87c335be835edc924233ea64792\": container with ID starting with ae16916df4e5810195066ef678b2217175adf87c335be835edc924233ea64792 not found: ID does not exist" containerID="ae16916df4e5810195066ef678b2217175adf87c335be835edc924233ea64792" Nov 03 21:30:32.932774 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.932601 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae16916df4e5810195066ef678b2217175adf87c335be835edc924233ea64792"} err="failed to get container status \"ae16916df4e5810195066ef678b2217175adf87c335be835edc924233ea64792\": rpc error: code = NotFound desc = could not find container \"ae16916df4e5810195066ef678b2217175adf87c335be835edc924233ea64792\": container with ID starting with ae16916df4e5810195066ef678b2217175adf87c335be835edc924233ea64792 not found: ID does not exist" Nov 03 21:30:32.932774 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.932639 5041 scope.go:122] "RemoveContainer" containerID="e16de07c8a8d6c890d059e74651fd941023879c54134540c9620a847e99d1dfd" Nov 03 21:30:32.941161 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.941145 5041 scope.go:122] "RemoveContainer" containerID="e16de07c8a8d6c890d059e74651fd941023879c54134540c9620a847e99d1dfd" Nov 03 21:30:32.941392 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:30:32.941372 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16de07c8a8d6c890d059e74651fd941023879c54134540c9620a847e99d1dfd\": container with ID starting with e16de07c8a8d6c890d059e74651fd941023879c54134540c9620a847e99d1dfd not found: ID does not exist" containerID="e16de07c8a8d6c890d059e74651fd941023879c54134540c9620a847e99d1dfd" Nov 03 21:30:32.941477 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.941396 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16de07c8a8d6c890d059e74651fd941023879c54134540c9620a847e99d1dfd"} err="failed to get container status \"e16de07c8a8d6c890d059e74651fd941023879c54134540c9620a847e99d1dfd\": rpc error: code = NotFound desc = could not find container \"e16de07c8a8d6c890d059e74651fd941023879c54134540c9620a847e99d1dfd\": container with ID starting with e16de07c8a8d6c890d059e74651fd941023879c54134540c9620a847e99d1dfd not found: ID does not exist" Nov 03 21:30:32.944763 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.944741 5041 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/306f723b-4c8b-407b-80b9-26e1c1045c2f-service-ca-bundle\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:32.944821 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.944777 5041 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/306f723b-4c8b-407b-80b9-26e1c1045c2f-archives-path\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:32.944821 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.944787 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ktf5\" (UniqueName: \"kubernetes.io/projected/ba86e754-0fcc-46ab-a5e2-10609342eb13-kube-api-access-5ktf5\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:32.944821 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.944795 5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/306f723b-4c8b-407b-80b9-26e1c1045c2f-trusted-ca-bundle\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:32.944821 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.944804 5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba86e754-0fcc-46ab-a5e2-10609342eb13-trusted-ca-bundle\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:32.944821 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.944812 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drjm5\" (UniqueName: \"kubernetes.io/projected/306f723b-4c8b-407b-80b9-26e1c1045c2f-kube-api-access-drjm5\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:32.976062 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.976041 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-pvgwb-whz8p"] Nov 03 21:30:32.979257 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:32.979239 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-pvgwb-whz8p"] Nov 03 21:30:33.014458 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:33.014437 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-dflhb-2sf79"] Nov 03 21:30:33.020040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:33.020021 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-dflhb-2sf79"] Nov 03 21:30:34.619215 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:34.619183 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="306f723b-4c8b-407b-80b9-26e1c1045c2f" path="/var/lib/kubelet/pods/306f723b-4c8b-407b-80b9-26e1c1045c2f/volumes" Nov 03 21:30:34.619576 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:34.619566 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba86e754-0fcc-46ab-a5e2-10609342eb13" path="/var/lib/kubelet/pods/ba86e754-0fcc-46ab-a5e2-10609342eb13/volumes" Nov 03 21:30:40.774782 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:30:40.774743 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:30:41.578198 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.578155 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" podUID="a6467adf-c26e-46ae-a212-98c9846bfe0e" containerName="registry" containerID="cri-o://c58ae1a7bbeb222f9ee0c711cccf377ee225617e4841defef85d38f67f6415ba" gracePeriod=30 Nov 03 21:30:41.816141 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.816113 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:30:41.917765 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.917697 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-image-registry-private-configuration\") pod \"a6467adf-c26e-46ae-a212-98c9846bfe0e\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " Nov 03 21:30:41.917765 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.917742 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-trusted-ca\") pod \"a6467adf-c26e-46ae-a212-98c9846bfe0e\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " Nov 03 21:30:41.917965 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.917775 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-certificates\") pod \"a6467adf-c26e-46ae-a212-98c9846bfe0e\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " Nov 03 21:30:41.917965 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.917806 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-bound-sa-token\") pod \"a6467adf-c26e-46ae-a212-98c9846bfe0e\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " Nov 03 21:30:41.917965 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.917823 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-kube-api-access-sldfp\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-kube-api-access-sldfp\") pod \"a6467adf-c26e-46ae-a212-98c9846bfe0e\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " Nov 03 21:30:41.917965 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.917856 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-installation-pull-secrets\") pod \"a6467adf-c26e-46ae-a212-98c9846bfe0e\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " Nov 03 21:30:41.917965 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.917903 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls\") pod \"a6467adf-c26e-46ae-a212-98c9846bfe0e\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " Nov 03 21:30:41.917965 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.917940 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6467adf-c26e-46ae-a212-98c9846bfe0e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6467adf-c26e-46ae-a212-98c9846bfe0e-ca-trust-extracted\") pod \"a6467adf-c26e-46ae-a212-98c9846bfe0e\" (UID: \"a6467adf-c26e-46ae-a212-98c9846bfe0e\") " Nov 03 21:30:41.918252 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.918206 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-certificates" pod "a6467adf-c26e-46ae-a212-98c9846bfe0e" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:41.918592 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.918561 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-trusted-ca" pod "a6467adf-c26e-46ae-a212-98c9846bfe0e" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:41.920383 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.920344 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-image-registry-private-configuration" pod "a6467adf-c26e-46ae-a212-98c9846bfe0e" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:30:41.920522 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.920445 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-bound-sa-token" pod "a6467adf-c26e-46ae-a212-98c9846bfe0e" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:41.920704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.920646 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-installation-pull-secrets" pod "a6467adf-c26e-46ae-a212-98c9846bfe0e" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:30:41.920704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.920694 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-kube-api-access-sldfp" pod "a6467adf-c26e-46ae-a212-98c9846bfe0e" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e"). InnerVolumeSpecName "kube-api-access-sldfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:41.921781 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.921754 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls" pod "a6467adf-c26e-46ae-a212-98c9846bfe0e" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:41.925038 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.925013 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6467adf-c26e-46ae-a212-98c9846bfe0e-ca-trust-extracted" pod "a6467adf-c26e-46ae-a212-98c9846bfe0e" (UID: "a6467adf-c26e-46ae-a212-98c9846bfe0e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:30:41.961556 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.961524 5041 generic.go:356] "Generic (PLEG): container finished" podID="a6467adf-c26e-46ae-a212-98c9846bfe0e" containerID="c58ae1a7bbeb222f9ee0c711cccf377ee225617e4841defef85d38f67f6415ba" exitCode=0 Nov 03 21:30:41.961803 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.961588 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" Nov 03 21:30:41.961803 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.961609 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" event={"ID":"a6467adf-c26e-46ae-a212-98c9846bfe0e","Type":"ContainerDied","Data":"c58ae1a7bbeb222f9ee0c711cccf377ee225617e4841defef85d38f67f6415ba"} Nov 03 21:30:41.961803 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.961680 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bf564b6d-c6gsv" event={"ID":"a6467adf-c26e-46ae-a212-98c9846bfe0e","Type":"ContainerDied","Data":"a2cbdef9537061f75df5f3e4a5f302627722eeb4999920ceabe193f9cca1397e"} Nov 03 21:30:41.961803 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.961685 5041 scope.go:122] "RemoveContainer" containerID="c58ae1a7bbeb222f9ee0c711cccf377ee225617e4841defef85d38f67f6415ba" Nov 03 21:30:41.969406 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.969389 5041 scope.go:122] "RemoveContainer" containerID="c58ae1a7bbeb222f9ee0c711cccf377ee225617e4841defef85d38f67f6415ba" Nov 03 21:30:41.969697 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:30:41.969675 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58ae1a7bbeb222f9ee0c711cccf377ee225617e4841defef85d38f67f6415ba\": container with ID starting with c58ae1a7bbeb222f9ee0c711cccf377ee225617e4841defef85d38f67f6415ba not found: ID does not exist" containerID="c58ae1a7bbeb222f9ee0c711cccf377ee225617e4841defef85d38f67f6415ba" Nov 03 21:30:41.969771 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:41.969705 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58ae1a7bbeb222f9ee0c711cccf377ee225617e4841defef85d38f67f6415ba"} err="failed to get container status \"c58ae1a7bbeb222f9ee0c711cccf377ee225617e4841defef85d38f67f6415ba\": rpc error: code = NotFound desc = could not find container \"c58ae1a7bbeb222f9ee0c711cccf377ee225617e4841defef85d38f67f6415ba\": container with ID starting with c58ae1a7bbeb222f9ee0c711cccf377ee225617e4841defef85d38f67f6415ba not found: ID does not exist" Nov 03 21:30:42.015948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:42.015918 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bf564b6d-c6gsv"] Nov 03 21:30:42.018527 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:42.018500 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sldfp\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-kube-api-access-sldfp\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:42.018629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:42.018532 5041 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-installation-pull-secrets\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:42.018629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:42.018548 5041 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-tls\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:42.018629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:42.018562 5041 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6467adf-c26e-46ae-a212-98c9846bfe0e-ca-trust-extracted\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:42.018629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:42.018577 5041 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6467adf-c26e-46ae-a212-98c9846bfe0e-image-registry-private-configuration\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:42.018629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:42.018591 5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-trusted-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:42.018629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:42.018604 5041 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6467adf-c26e-46ae-a212-98c9846bfe0e-registry-certificates\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:42.018629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:42.018617 5041 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6467adf-c26e-46ae-a212-98c9846bfe0e-bound-sa-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:30:42.019145 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:42.019128 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7bf564b6d-c6gsv"] Nov 03 21:30:42.618350 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:30:42.618315 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6467adf-c26e-46ae-a212-98c9846bfe0e" path="/var/lib/kubelet/pods/a6467adf-c26e-46ae-a212-98c9846bfe0e/volumes" Nov 03 21:31:00.026702 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:31:00.026652 5041 generic.go:356] "Generic (PLEG): container finished" podID="37a8184c-e9f5-4d96-a816-e0191c32ed06" containerID="e0b15d83f86ee8174c763d961444733403907502b2abde73dc05af24a76e6bc5" exitCode=0 Nov 03 21:31:00.026702 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:31:00.026690 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" event={"ID":"37a8184c-e9f5-4d96-a816-e0191c32ed06","Type":"ContainerDied","Data":"e0b15d83f86ee8174c763d961444733403907502b2abde73dc05af24a76e6bc5"} Nov 03 21:31:00.027127 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:31:00.026996 5041 scope.go:122] "RemoveContainer" containerID="e0b15d83f86ee8174c763d961444733403907502b2abde73dc05af24a76e6bc5" Nov 03 21:31:01.032251 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:31:01.032217 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-r9qwk" event={"ID":"37a8184c-e9f5-4d96-a816-e0191c32ed06","Type":"ContainerStarted","Data":"24b0aad0f5059f560ac496b29e61498f6f182c66e27ab6e10fd9dd7dcaa4738d"} Nov 03 21:31:10.774975 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:31:10.774938 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:31:40.776001 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:31:40.775963 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:32:10.775599 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:32:10.775514 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:32:40.775977 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:32:40.775938 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:33:10.775830 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:33:10.775792 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:33:40.776246 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:33:40.776162 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:34:10.777749 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:34:10.777705 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:34:40.776168 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:34:40.776134 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:34:44.634579 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:34:44.634548 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 21:34:44.637997 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:34:44.637875 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 21:35:10.762633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.762544 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr"] Nov 03 21:35:10.763010 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.762911 5041 cpu_manager.go:414] "containerMap: removing container" podUID="cd1064c9-bade-4061-9b89-5d15e6a26a3d" containerName="registry" Nov 03 21:35:10.763010 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.762925 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd1064c9-bade-4061-9b89-5d15e6a26a3d" containerName="registry" Nov 03 21:35:10.763010 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.762935 5041 cpu_manager.go:414] "containerMap: removing container" podUID="a6467adf-c26e-46ae-a212-98c9846bfe0e" containerName="registry" Nov 03 21:35:10.763010 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.762940 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6467adf-c26e-46ae-a212-98c9846bfe0e" containerName="registry" Nov 03 21:35:10.763010 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.762948 5041 cpu_manager.go:414] "containerMap: removing container" podUID="ba86e754-0fcc-46ab-a5e2-10609342eb13" containerName="insights-gathering" Nov 03 21:35:10.763010 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.762954 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba86e754-0fcc-46ab-a5e2-10609342eb13" containerName="insights-gathering" Nov 03 21:35:10.763010 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.762966 5041 cpu_manager.go:414] "containerMap: removing container" podUID="306f723b-4c8b-407b-80b9-26e1c1045c2f" containerName="insights-gathering" Nov 03 21:35:10.763010 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.762971 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306f723b-4c8b-407b-80b9-26e1c1045c2f" containerName="insights-gathering" Nov 03 21:35:10.763247 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.763030 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd1064c9-bade-4061-9b89-5d15e6a26a3d" containerName="registry" Nov 03 21:35:10.763247 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.763037 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6467adf-c26e-46ae-a212-98c9846bfe0e" containerName="registry" Nov 03 21:35:10.763247 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.763042 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba86e754-0fcc-46ab-a5e2-10609342eb13" containerName="insights-gathering" Nov 03 21:35:10.763247 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.763049 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="306f723b-4c8b-407b-80b9-26e1c1045c2f" containerName="insights-gathering" Nov 03 21:35:10.765500 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.765478 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" Nov 03 21:35:10.776892 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:35:10.776865 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:35:10.783618 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.783589 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-7tgs7\"" Nov 03 21:35:10.797793 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.797772 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 21:35:10.797902 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.797799 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 21:35:10.798897 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.798879 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr"] Nov 03 21:35:10.827967 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.827940 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f25c1185-d4ca-4a23-b08c-0988faccc447-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr\" (UID: \"f25c1185-d4ca-4a23-b08c-0988faccc447\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" Nov 03 21:35:10.828079 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.827980 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f25c1185-d4ca-4a23-b08c-0988faccc447-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr\" (UID: \"f25c1185-d4ca-4a23-b08c-0988faccc447\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" Nov 03 21:35:10.828123 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.828100 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbn67\" (UniqueName: \"kubernetes.io/projected/f25c1185-d4ca-4a23-b08c-0988faccc447-kube-api-access-fbn67\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr\" (UID: \"f25c1185-d4ca-4a23-b08c-0988faccc447\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" Nov 03 21:35:10.861455 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.861425 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62"] Nov 03 21:35:10.864495 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.864476 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" Nov 03 21:35:10.883597 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.883568 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62"] Nov 03 21:35:10.909954 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.909924 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-zmkbq\"" Nov 03 21:35:10.929317 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.929285 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f25c1185-d4ca-4a23-b08c-0988faccc447-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr\" (UID: \"f25c1185-d4ca-4a23-b08c-0988faccc447\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" Nov 03 21:35:10.929441 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.929338 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f25c1185-d4ca-4a23-b08c-0988faccc447-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr\" (UID: \"f25c1185-d4ca-4a23-b08c-0988faccc447\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" Nov 03 21:35:10.929441 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.929384 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxl8w\" (UniqueName: \"kubernetes.io/projected/411326e6-c41f-4ba7-b116-b909f9edd775-kube-api-access-hxl8w\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62\" (UID: \"411326e6-c41f-4ba7-b116-b909f9edd775\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" Nov 03 21:35:10.929441 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.929418 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/411326e6-c41f-4ba7-b116-b909f9edd775-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62\" (UID: \"411326e6-c41f-4ba7-b116-b909f9edd775\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" Nov 03 21:35:10.929575 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.929465 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/411326e6-c41f-4ba7-b116-b909f9edd775-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62\" (UID: \"411326e6-c41f-4ba7-b116-b909f9edd775\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" Nov 03 21:35:10.929575 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.929561 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbn67\" (UniqueName: \"kubernetes.io/projected/f25c1185-d4ca-4a23-b08c-0988faccc447-kube-api-access-fbn67\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr\" (UID: \"f25c1185-d4ca-4a23-b08c-0988faccc447\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" Nov 03 21:35:10.929715 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.929699 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f25c1185-d4ca-4a23-b08c-0988faccc447-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr\" (UID: \"f25c1185-d4ca-4a23-b08c-0988faccc447\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" Nov 03 21:35:10.929755 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.929735 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f25c1185-d4ca-4a23-b08c-0988faccc447-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr\" (UID: \"f25c1185-d4ca-4a23-b08c-0988faccc447\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" Nov 03 21:35:10.953326 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.953301 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbn67\" (UniqueName: \"kubernetes.io/projected/f25c1185-d4ca-4a23-b08c-0988faccc447-kube-api-access-fbn67\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr\" (UID: \"f25c1185-d4ca-4a23-b08c-0988faccc447\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" Nov 03 21:35:10.960901 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.960877 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj"] Nov 03 21:35:10.964132 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.964116 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" Nov 03 21:35:10.982156 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:10.982134 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj"] Nov 03 21:35:11.010077 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.010052 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-gfr6x\"" Nov 03 21:35:11.030458 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.030394 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e710705-b22a-470d-9cd0-15308632b744-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj\" (UID: \"8e710705-b22a-470d-9cd0-15308632b744\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" Nov 03 21:35:11.030458 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.030432 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e710705-b22a-470d-9cd0-15308632b744-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj\" (UID: \"8e710705-b22a-470d-9cd0-15308632b744\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" Nov 03 21:35:11.030575 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.030514 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxl8w\" (UniqueName: \"kubernetes.io/projected/411326e6-c41f-4ba7-b116-b909f9edd775-kube-api-access-hxl8w\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62\" (UID: \"411326e6-c41f-4ba7-b116-b909f9edd775\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" Nov 03 21:35:11.030575 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.030551 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/411326e6-c41f-4ba7-b116-b909f9edd775-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62\" (UID: \"411326e6-c41f-4ba7-b116-b909f9edd775\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" Nov 03 21:35:11.030694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.030578 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ssbm\" (UniqueName: \"kubernetes.io/projected/8e710705-b22a-470d-9cd0-15308632b744-kube-api-access-5ssbm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj\" (UID: \"8e710705-b22a-470d-9cd0-15308632b744\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" Nov 03 21:35:11.030694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.030616 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/411326e6-c41f-4ba7-b116-b909f9edd775-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62\" (UID: \"411326e6-c41f-4ba7-b116-b909f9edd775\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" Nov 03 21:35:11.030957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.030939 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/411326e6-c41f-4ba7-b116-b909f9edd775-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62\" (UID: \"411326e6-c41f-4ba7-b116-b909f9edd775\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" Nov 03 21:35:11.031005 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.030945 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/411326e6-c41f-4ba7-b116-b909f9edd775-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62\" (UID: \"411326e6-c41f-4ba7-b116-b909f9edd775\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" Nov 03 21:35:11.050718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.050691 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxl8w\" (UniqueName: \"kubernetes.io/projected/411326e6-c41f-4ba7-b116-b909f9edd775-kube-api-access-hxl8w\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62\" (UID: \"411326e6-c41f-4ba7-b116-b909f9edd775\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" Nov 03 21:35:11.061928 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.061909 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9"] Nov 03 21:35:11.065171 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.065156 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" Nov 03 21:35:11.081444 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.081421 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" Nov 03 21:35:11.083682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.083645 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9"] Nov 03 21:35:11.110973 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.110945 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-fmzsn\"" Nov 03 21:35:11.131515 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.131123 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f3f49aa-e369-46c3-8e35-5749f848114d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9\" (UID: \"5f3f49aa-e369-46c3-8e35-5749f848114d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" Nov 03 21:35:11.131515 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.131161 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ssbm\" (UniqueName: \"kubernetes.io/projected/8e710705-b22a-470d-9cd0-15308632b744-kube-api-access-5ssbm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj\" (UID: \"8e710705-b22a-470d-9cd0-15308632b744\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" Nov 03 21:35:11.131515 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.131203 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq97g\" (UniqueName: \"kubernetes.io/projected/5f3f49aa-e369-46c3-8e35-5749f848114d-kube-api-access-cq97g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9\" (UID: \"5f3f49aa-e369-46c3-8e35-5749f848114d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" Nov 03 21:35:11.131515 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.131236 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e710705-b22a-470d-9cd0-15308632b744-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj\" (UID: \"8e710705-b22a-470d-9cd0-15308632b744\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" Nov 03 21:35:11.131515 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.131281 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f3f49aa-e369-46c3-8e35-5749f848114d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9\" (UID: \"5f3f49aa-e369-46c3-8e35-5749f848114d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" Nov 03 21:35:11.131515 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.131307 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e710705-b22a-470d-9cd0-15308632b744-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj\" (UID: \"8e710705-b22a-470d-9cd0-15308632b744\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" Nov 03 21:35:11.132289 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.131710 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e710705-b22a-470d-9cd0-15308632b744-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj\" (UID: \"8e710705-b22a-470d-9cd0-15308632b744\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" Nov 03 21:35:11.132289 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.131959 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e710705-b22a-470d-9cd0-15308632b744-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj\" (UID: \"8e710705-b22a-470d-9cd0-15308632b744\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" Nov 03 21:35:11.153353 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.153327 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ssbm\" (UniqueName: \"kubernetes.io/projected/8e710705-b22a-470d-9cd0-15308632b744-kube-api-access-5ssbm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj\" (UID: \"8e710705-b22a-470d-9cd0-15308632b744\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" Nov 03 21:35:11.178870 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.178845 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" Nov 03 21:35:11.232151 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.232114 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f3f49aa-e369-46c3-8e35-5749f848114d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9\" (UID: \"5f3f49aa-e369-46c3-8e35-5749f848114d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" Nov 03 21:35:11.232286 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.232209 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq97g\" (UniqueName: \"kubernetes.io/projected/5f3f49aa-e369-46c3-8e35-5749f848114d-kube-api-access-cq97g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9\" (UID: \"5f3f49aa-e369-46c3-8e35-5749f848114d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" Nov 03 21:35:11.232348 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.232280 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f3f49aa-e369-46c3-8e35-5749f848114d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9\" (UID: \"5f3f49aa-e369-46c3-8e35-5749f848114d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" Nov 03 21:35:11.233211 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.232584 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f3f49aa-e369-46c3-8e35-5749f848114d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9\" (UID: \"5f3f49aa-e369-46c3-8e35-5749f848114d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" Nov 03 21:35:11.233211 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.232639 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f3f49aa-e369-46c3-8e35-5749f848114d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9\" (UID: \"5f3f49aa-e369-46c3-8e35-5749f848114d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" Nov 03 21:35:11.253798 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.253769 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq97g\" (UniqueName: \"kubernetes.io/projected/5f3f49aa-e369-46c3-8e35-5749f848114d-kube-api-access-cq97g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9\" (UID: \"5f3f49aa-e369-46c3-8e35-5749f848114d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" Nov 03 21:35:11.279372 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.279346 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" Nov 03 21:35:11.375165 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.375134 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr"] Nov 03 21:35:11.378157 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:35:11.378129 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25c1185_d4ca_4a23_b08c_0988faccc447.slice/crio-b5ec2421e423d0b000f288a434c1dbfe541ef6381d5d3daa78c9172cc20cfe0d WatchSource:0}: Error finding container b5ec2421e423d0b000f288a434c1dbfe541ef6381d5d3daa78c9172cc20cfe0d: Status 404 returned error can't find the container with id b5ec2421e423d0b000f288a434c1dbfe541ef6381d5d3daa78c9172cc20cfe0d Nov 03 21:35:11.379925 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.379904 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" Nov 03 21:35:11.380045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.380027 5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:35:11.472426 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.472395 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62"] Nov 03 21:35:11.475494 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:35:11.475472 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411326e6_c41f_4ba7_b116_b909f9edd775.slice/crio-f3a57f3910abed10242382500c82582bb029704f9e6f668fb96410019cb4c3df WatchSource:0}: Error finding container f3a57f3910abed10242382500c82582bb029704f9e6f668fb96410019cb4c3df: Status 404 returned error can't find the container with id f3a57f3910abed10242382500c82582bb029704f9e6f668fb96410019cb4c3df Nov 03 21:35:11.570971 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.570899 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj"] Nov 03 21:35:11.574227 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:35:11.574201 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e710705_b22a_470d_9cd0_15308632b744.slice/crio-94120c47ae2c625939dfd3701207a09ee60c2550717e3d348817ce5d84f347b8 WatchSource:0}: Error finding container 94120c47ae2c625939dfd3701207a09ee60c2550717e3d348817ce5d84f347b8: Status 404 returned error can't find the container with id 94120c47ae2c625939dfd3701207a09ee60c2550717e3d348817ce5d84f347b8 Nov 03 21:35:11.669708 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.669672 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9"] Nov 03 21:35:11.673403 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:35:11.673366 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3f49aa_e369_46c3_8e35_5749f848114d.slice/crio-5375153aae4a2157d50783cb9bf9d1eb8cbf1ab30d457308220125b12e3c98fc WatchSource:0}: Error finding container 5375153aae4a2157d50783cb9bf9d1eb8cbf1ab30d457308220125b12e3c98fc: Status 404 returned error can't find the container with id 5375153aae4a2157d50783cb9bf9d1eb8cbf1ab30d457308220125b12e3c98fc Nov 03 21:35:11.960776 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.960742 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm"] Nov 03 21:35:11.964403 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.964376 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" Nov 03 21:35:11.981489 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.981461 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-sr6kp\"" Nov 03 21:35:11.983033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.983005 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm"] Nov 03 21:35:11.995635 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.995605 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" event={"ID":"f25c1185-d4ca-4a23-b08c-0988faccc447","Type":"ContainerStarted","Data":"b5ec2421e423d0b000f288a434c1dbfe541ef6381d5d3daa78c9172cc20cfe0d"} Nov 03 21:35:11.997265 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.997238 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" event={"ID":"8e710705-b22a-470d-9cd0-15308632b744","Type":"ContainerStarted","Data":"94120c47ae2c625939dfd3701207a09ee60c2550717e3d348817ce5d84f347b8"} Nov 03 21:35:11.999127 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:11.999105 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" event={"ID":"411326e6-c41f-4ba7-b116-b909f9edd775","Type":"ContainerStarted","Data":"f3a57f3910abed10242382500c82582bb029704f9e6f668fb96410019cb4c3df"} Nov 03 21:35:12.000553 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:12.000531 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" event={"ID":"5f3f49aa-e369-46c3-8e35-5749f848114d","Type":"ContainerStarted","Data":"5375153aae4a2157d50783cb9bf9d1eb8cbf1ab30d457308220125b12e3c98fc"} Nov 03 21:35:12.044581 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:12.044545 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/231f4e89-43ae-4acd-a012-405fc60841c6-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm\" (UID: \"231f4e89-43ae-4acd-a012-405fc60841c6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" Nov 03 21:35:12.044779 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:12.044594 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm4ph\" (UniqueName: \"kubernetes.io/projected/231f4e89-43ae-4acd-a012-405fc60841c6-kube-api-access-sm4ph\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm\" (UID: \"231f4e89-43ae-4acd-a012-405fc60841c6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" Nov 03 21:35:12.044779 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:12.044633 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/231f4e89-43ae-4acd-a012-405fc60841c6-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm\" (UID: \"231f4e89-43ae-4acd-a012-405fc60841c6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" Nov 03 21:35:12.145219 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:12.145182 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/231f4e89-43ae-4acd-a012-405fc60841c6-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm\" (UID: \"231f4e89-43ae-4acd-a012-405fc60841c6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" Nov 03 21:35:12.145391 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:12.145242 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm4ph\" (UniqueName: \"kubernetes.io/projected/231f4e89-43ae-4acd-a012-405fc60841c6-kube-api-access-sm4ph\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm\" (UID: \"231f4e89-43ae-4acd-a012-405fc60841c6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" Nov 03 21:35:12.145391 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:12.145302 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/231f4e89-43ae-4acd-a012-405fc60841c6-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm\" (UID: \"231f4e89-43ae-4acd-a012-405fc60841c6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" Nov 03 21:35:12.145773 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:12.145749 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/231f4e89-43ae-4acd-a012-405fc60841c6-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm\" (UID: \"231f4e89-43ae-4acd-a012-405fc60841c6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" Nov 03 21:35:12.146003 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:12.145982 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/231f4e89-43ae-4acd-a012-405fc60841c6-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm\" (UID: \"231f4e89-43ae-4acd-a012-405fc60841c6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" Nov 03 21:35:12.175383 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:12.175347 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm4ph\" (UniqueName: \"kubernetes.io/projected/231f4e89-43ae-4acd-a012-405fc60841c6-kube-api-access-sm4ph\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm\" (UID: \"231f4e89-43ae-4acd-a012-405fc60841c6\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" Nov 03 21:35:12.294942 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:12.294862 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" Nov 03 21:35:12.609368 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:12.609286 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm"] Nov 03 21:35:12.619930 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:35:12.619779 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231f4e89_43ae_4acd_a012_405fc60841c6.slice/crio-d696b010a2f9e18fd8891d2a143636b48720e26d46c0651a745332e93c50afd4 WatchSource:0}: Error finding container d696b010a2f9e18fd8891d2a143636b48720e26d46c0651a745332e93c50afd4: Status 404 returned error can't find the container with id d696b010a2f9e18fd8891d2a143636b48720e26d46c0651a745332e93c50afd4 Nov 03 21:35:13.013925 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:13.013835 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" event={"ID":"231f4e89-43ae-4acd-a012-405fc60841c6","Type":"ContainerStarted","Data":"d696b010a2f9e18fd8891d2a143636b48720e26d46c0651a745332e93c50afd4"} Nov 03 21:35:13.888376 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:13.888340 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv"] Nov 03 21:35:13.906675 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:13.906626 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv"] Nov 03 21:35:13.906861 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:13.906840 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" Nov 03 21:35:13.936778 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:13.936742 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-vxkch\"" Nov 03 21:35:14.063580 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:14.063543 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9956d082-d0ee-459d-b604-1535b45a4e15-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv\" (UID: \"9956d082-d0ee-459d-b604-1535b45a4e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" Nov 03 21:35:14.063978 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:14.063647 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9956d082-d0ee-459d-b604-1535b45a4e15-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv\" (UID: \"9956d082-d0ee-459d-b604-1535b45a4e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" Nov 03 21:35:14.063978 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:14.063704 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g59vb\" (UniqueName: \"kubernetes.io/projected/9956d082-d0ee-459d-b604-1535b45a4e15-kube-api-access-g59vb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv\" (UID: \"9956d082-d0ee-459d-b604-1535b45a4e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" Nov 03 21:35:14.164636 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:14.164547 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9956d082-d0ee-459d-b604-1535b45a4e15-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv\" (UID: \"9956d082-d0ee-459d-b604-1535b45a4e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" Nov 03 21:35:14.164829 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:14.164656 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9956d082-d0ee-459d-b604-1535b45a4e15-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv\" (UID: \"9956d082-d0ee-459d-b604-1535b45a4e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" Nov 03 21:35:14.164829 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:14.164705 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g59vb\" (UniqueName: \"kubernetes.io/projected/9956d082-d0ee-459d-b604-1535b45a4e15-kube-api-access-g59vb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv\" (UID: \"9956d082-d0ee-459d-b604-1535b45a4e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" Nov 03 21:35:14.165184 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:14.165130 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9956d082-d0ee-459d-b604-1535b45a4e15-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv\" (UID: \"9956d082-d0ee-459d-b604-1535b45a4e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" Nov 03 21:35:14.165308 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:14.165224 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9956d082-d0ee-459d-b604-1535b45a4e15-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv\" (UID: \"9956d082-d0ee-459d-b604-1535b45a4e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" Nov 03 21:35:14.203503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:14.203460 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g59vb\" (UniqueName: \"kubernetes.io/projected/9956d082-d0ee-459d-b604-1535b45a4e15-kube-api-access-g59vb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv\" (UID: \"9956d082-d0ee-459d-b604-1535b45a4e15\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" Nov 03 21:35:14.228408 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:14.228376 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" Nov 03 21:35:14.577049 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:14.577013 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv"] Nov 03 21:35:14.583464 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:35:14.583430 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9956d082_d0ee_459d_b604_1535b45a4e15.slice/crio-f743f5dcf5d6ed197cda9b8b045e71dbdfc71ff9e3d5606f5b82b38a14ea1f64 WatchSource:0}: Error finding container f743f5dcf5d6ed197cda9b8b045e71dbdfc71ff9e3d5606f5b82b38a14ea1f64: Status 404 returned error can't find the container with id f743f5dcf5d6ed197cda9b8b045e71dbdfc71ff9e3d5606f5b82b38a14ea1f64 Nov 03 21:35:15.029393 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.029305 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" event={"ID":"9956d082-d0ee-459d-b604-1535b45a4e15","Type":"ContainerStarted","Data":"f743f5dcf5d6ed197cda9b8b045e71dbdfc71ff9e3d5606f5b82b38a14ea1f64"} Nov 03 21:35:15.188326 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.186655 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf"] Nov 03 21:35:15.204561 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.203922 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.219436 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.219407 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf"] Nov 03 21:35:15.230228 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.230136 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pgck4\"" Nov 03 21:35:15.230228 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.230201 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 21:35:15.274178 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.273584 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.274178 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.273647 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.274178 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.273708 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.274178 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.273734 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.274178 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.273764 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjjg6\" (UniqueName: \"kubernetes.io/projected/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-kube-api-access-qjjg6\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.274178 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.273835 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.274178 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.273869 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.274178 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.273903 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.274178 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.273934 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.316742 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.315030 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8"] Nov 03 21:35:15.330087 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.328409 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.343088 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.340706 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8"] Nov 03 21:35:15.346045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.345856 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-c9v2f\"" Nov 03 21:35:15.375495 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.375441 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.375495 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.375491 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.375886 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.375528 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.375886 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.375559 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.375886 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.375590 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.375886 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.375632 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.375886 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.375687 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.375886 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.375713 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.375886 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.375745 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjjg6\" (UniqueName: \"kubernetes.io/projected/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-kube-api-access-qjjg6\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.376565 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.376538 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.376800 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.376776 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.377242 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.377217 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.377537 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.377514 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.377627 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.377574 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.379723 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.379701 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.382905 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.382881 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.399337 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.399267 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.399995 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.399957 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjjg6\" (UniqueName: \"kubernetes.io/projected/bd662fa8-196b-4cad-b414-1bc2dbd68aa1-kube-api-access-qjjg6\") pod \"maas-default-gateway-openshift-default-75d7c785f-6kgrf\" (UID: \"bd662fa8-196b-4cad-b414-1bc2dbd68aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.476647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.476569 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d77b6f1-de97-4e82-9fb6-0735930827a2-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.476864 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.476722 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d77b6f1-de97-4e82-9fb6-0735930827a2-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.476864 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.476765 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d77b6f1-de97-4e82-9fb6-0735930827a2-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.477361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.476863 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d77b6f1-de97-4e82-9fb6-0735930827a2-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.477361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.476899 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d77b6f1-de97-4e82-9fb6-0735930827a2-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.477361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.477270 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc5q6\" (UniqueName: \"kubernetes.io/projected/1d77b6f1-de97-4e82-9fb6-0735930827a2-kube-api-access-cc5q6\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.477361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.477307 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d77b6f1-de97-4e82-9fb6-0735930827a2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.477361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.477403 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d77b6f1-de97-4e82-9fb6-0735930827a2-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.477814 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.477508 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d77b6f1-de97-4e82-9fb6-0735930827a2-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.537601 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.537567 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:15.579382 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.578790 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d77b6f1-de97-4e82-9fb6-0735930827a2-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.579382 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.578844 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d77b6f1-de97-4e82-9fb6-0735930827a2-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.579382 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.579315 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d77b6f1-de97-4e82-9fb6-0735930827a2-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.579382 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.579363 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d77b6f1-de97-4e82-9fb6-0735930827a2-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.580508 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.579924 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d77b6f1-de97-4e82-9fb6-0735930827a2-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.580508 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.579975 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d77b6f1-de97-4e82-9fb6-0735930827a2-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.580508 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.580003 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d77b6f1-de97-4e82-9fb6-0735930827a2-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.580508 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.580053 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d77b6f1-de97-4e82-9fb6-0735930827a2-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.580508 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.580087 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d77b6f1-de97-4e82-9fb6-0735930827a2-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.580508 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.580127 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc5q6\" (UniqueName: \"kubernetes.io/projected/1d77b6f1-de97-4e82-9fb6-0735930827a2-kube-api-access-cc5q6\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.580508 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.580156 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d77b6f1-de97-4e82-9fb6-0735930827a2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.582841 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.582108 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d77b6f1-de97-4e82-9fb6-0735930827a2-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.582841 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.582058 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d77b6f1-de97-4e82-9fb6-0735930827a2-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.583501 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.583477 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d77b6f1-de97-4e82-9fb6-0735930827a2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.591696 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.591623 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d77b6f1-de97-4e82-9fb6-0735930827a2-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.594370 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.594346 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d77b6f1-de97-4e82-9fb6-0735930827a2-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.602584 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.602557 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d77b6f1-de97-4e82-9fb6-0735930827a2-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.608514 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.608489 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc5q6\" (UniqueName: \"kubernetes.io/projected/1d77b6f1-de97-4e82-9fb6-0735930827a2-kube-api-access-cc5q6\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8\" (UID: \"1d77b6f1-de97-4e82-9fb6-0735930827a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.657150 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.656575 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:15.887099 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:15.886941 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf"] Nov 03 21:35:16.016373 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:16.016336 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8"] Nov 03 21:35:16.057628 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:16.057514 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" event={"ID":"1d77b6f1-de97-4e82-9fb6-0735930827a2","Type":"ContainerStarted","Data":"71752dd50cad9a64b2f49f8a6fd5cbcc7d6e7ab180d5f6564e9b3da93c8fea4f"} Nov 03 21:35:16.059797 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:16.059767 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" event={"ID":"bd662fa8-196b-4cad-b414-1bc2dbd68aa1","Type":"ContainerStarted","Data":"bcbd276d58e37dcb0666abefb5db1239e645744038636fac716d60dfb34728b1"} Nov 03 21:35:18.121964 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.121924 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6"] Nov 03 21:35:18.125724 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.125700 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" Nov 03 21:35:18.144047 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.144016 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6"] Nov 03 21:35:18.160346 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.158275 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq"] Nov 03 21:35:18.161493 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.161470 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" Nov 03 21:35:18.168512 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.168369 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq"] Nov 03 21:35:18.209417 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.209213 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a543bf82-126c-463e-9c78-d9353a1f69d8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6\" (UID: \"a543bf82-126c-463e-9c78-d9353a1f69d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" Nov 03 21:35:18.209417 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.209266 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a543bf82-126c-463e-9c78-d9353a1f69d8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6\" (UID: \"a543bf82-126c-463e-9c78-d9353a1f69d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" Nov 03 21:35:18.209417 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.209310 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwmp\" (UniqueName: \"kubernetes.io/projected/a543bf82-126c-463e-9c78-d9353a1f69d8-kube-api-access-qqwmp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6\" (UID: \"a543bf82-126c-463e-9c78-d9353a1f69d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" Nov 03 21:35:18.310915 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.310845 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sth5k\" (UniqueName: \"kubernetes.io/projected/8d360af4-8562-4b82-9c71-949a5183165b-kube-api-access-sth5k\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq\" (UID: \"8d360af4-8562-4b82-9c71-949a5183165b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" Nov 03 21:35:18.311091 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.310933 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a543bf82-126c-463e-9c78-d9353a1f69d8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6\" (UID: \"a543bf82-126c-463e-9c78-d9353a1f69d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" Nov 03 21:35:18.311091 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.310991 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a543bf82-126c-463e-9c78-d9353a1f69d8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6\" (UID: \"a543bf82-126c-463e-9c78-d9353a1f69d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" Nov 03 21:35:18.311091 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.311055 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwmp\" (UniqueName: \"kubernetes.io/projected/a543bf82-126c-463e-9c78-d9353a1f69d8-kube-api-access-qqwmp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6\" (UID: \"a543bf82-126c-463e-9c78-d9353a1f69d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" Nov 03 21:35:18.311290 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.311089 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d360af4-8562-4b82-9c71-949a5183165b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq\" (UID: \"8d360af4-8562-4b82-9c71-949a5183165b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" Nov 03 21:35:18.311290 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.311114 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d360af4-8562-4b82-9c71-949a5183165b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq\" (UID: \"8d360af4-8562-4b82-9c71-949a5183165b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" Nov 03 21:35:18.311596 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.311572 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a543bf82-126c-463e-9c78-d9353a1f69d8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6\" (UID: \"a543bf82-126c-463e-9c78-d9353a1f69d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" Nov 03 21:35:18.311871 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.311848 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a543bf82-126c-463e-9c78-d9353a1f69d8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6\" (UID: \"a543bf82-126c-463e-9c78-d9353a1f69d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" Nov 03 21:35:18.346479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.346408 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwmp\" (UniqueName: \"kubernetes.io/projected/a543bf82-126c-463e-9c78-d9353a1f69d8-kube-api-access-qqwmp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6\" (UID: \"a543bf82-126c-463e-9c78-d9353a1f69d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" Nov 03 21:35:18.413258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.412614 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d360af4-8562-4b82-9c71-949a5183165b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq\" (UID: \"8d360af4-8562-4b82-9c71-949a5183165b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" Nov 03 21:35:18.413258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.412689 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d360af4-8562-4b82-9c71-949a5183165b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq\" (UID: \"8d360af4-8562-4b82-9c71-949a5183165b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" Nov 03 21:35:18.413258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.412737 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sth5k\" (UniqueName: \"kubernetes.io/projected/8d360af4-8562-4b82-9c71-949a5183165b-kube-api-access-sth5k\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq\" (UID: \"8d360af4-8562-4b82-9c71-949a5183165b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" Nov 03 21:35:18.413258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.413156 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d360af4-8562-4b82-9c71-949a5183165b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq\" (UID: \"8d360af4-8562-4b82-9c71-949a5183165b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" Nov 03 21:35:18.413258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.413216 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d360af4-8562-4b82-9c71-949a5183165b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq\" (UID: \"8d360af4-8562-4b82-9c71-949a5183165b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" Nov 03 21:35:18.447655 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.447012 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" Nov 03 21:35:18.447655 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.447400 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sth5k\" (UniqueName: \"kubernetes.io/projected/8d360af4-8562-4b82-9c71-949a5183165b-kube-api-access-sth5k\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq\" (UID: \"8d360af4-8562-4b82-9c71-949a5183165b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" Nov 03 21:35:18.484098 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.484060 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" Nov 03 21:35:18.798158 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.797999 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6"] Nov 03 21:35:18.848613 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:18.848515 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq"] Nov 03 21:35:18.855020 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:35:18.854991 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d360af4_8562_4b82_9c71_949a5183165b.slice/crio-f1f8d331d1f929f5d102e7099e684bd99012fe476e87787ba713f471eb8201f1 WatchSource:0}: Error finding container f1f8d331d1f929f5d102e7099e684bd99012fe476e87787ba713f471eb8201f1: Status 404 returned error can't find the container with id f1f8d331d1f929f5d102e7099e684bd99012fe476e87787ba713f471eb8201f1 Nov 03 21:35:19.110434 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:19.110340 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" event={"ID":"a543bf82-126c-463e-9c78-d9353a1f69d8","Type":"ContainerStarted","Data":"fc442313f4917e71d9f590a55296721658a43301aad77dc2642a8f212f414548"} Nov 03 21:35:19.115556 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:19.115517 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" event={"ID":"8d360af4-8562-4b82-9c71-949a5183165b","Type":"ContainerStarted","Data":"f1f8d331d1f929f5d102e7099e684bd99012fe476e87787ba713f471eb8201f1"} Nov 03 21:35:24.801700 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:24.801193 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:35:24.801700 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:24.801289 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:35:24.801700 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:24.801339 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:35:24.825739 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:24.825696 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:35:24.825916 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:24.825791 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:35:24.825916 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:24.825832 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:35:25.216579 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:25.216533 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" event={"ID":"1d77b6f1-de97-4e82-9fb6-0735930827a2","Type":"ContainerStarted","Data":"7c0eda2dc53755a27200861b388bb3c9177da81872e4bb76a49851880b3acfab"} Nov 03 21:35:25.229047 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:25.229006 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" event={"ID":"bd662fa8-196b-4cad-b414-1bc2dbd68aa1","Type":"ContainerStarted","Data":"c7f3f2d62c1a8797934578723ff3a60a022c5fc9bda6e454cbbd61a2800f4047"} Nov 03 21:35:25.286683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:25.286593 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" podStartSLOduration=1.493598265 podStartE2EDuration="10.286576591s" totalImagesPullingTime="8.792978326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:25.285407191 +0000 UTC m=+340.792635508" watchObservedRunningTime="2026-11-03 21:35:25.286576591 +0000 UTC m=+340.793804876" Nov 03 21:35:25.329995 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:25.329637 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" podStartSLOduration=1.427841661 podStartE2EDuration="10.329616681s" totalImagesPullingTime="8.90177502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:25.328884682 +0000 UTC m=+340.836113000" watchObservedRunningTime="2026-11-03 21:35:25.329616681 +0000 UTC m=+340.836844962" Nov 03 21:35:25.538260 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:25.538112 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:25.544323 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:25.544024 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:25.657620 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:25.657576 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:25.663913 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:25.663879 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:26.239872 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:26.239825 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:26.240450 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:26.240049 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:26.240942 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:26.240920 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8" Nov 03 21:35:26.241520 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:26.241501 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:35:27.251492 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:27.250847 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" event={"ID":"f25c1185-d4ca-4a23-b08c-0988faccc447","Type":"ContainerDied","Data":"bbddb4ce24256950025de90c51885adcfc8cd5674fab5fe16478363da846da87"} Nov 03 21:35:27.251492 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:27.250817 5041 generic.go:356] "Generic (PLEG): container finished" podID="f25c1185-d4ca-4a23-b08c-0988faccc447" containerID="bbddb4ce24256950025de90c51885adcfc8cd5674fab5fe16478363da846da87" exitCode=0 Nov 03 21:35:28.259043 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:28.258992 5041 generic.go:356] "Generic (PLEG): container finished" podID="8e710705-b22a-470d-9cd0-15308632b744" containerID="7f23e4a5c630438e71ea73149fadf0acbb744516027a0a59605c13d8f5c53848" exitCode=0 Nov 03 21:35:28.259534 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:28.259096 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" event={"ID":"8e710705-b22a-470d-9cd0-15308632b744","Type":"ContainerDied","Data":"7f23e4a5c630438e71ea73149fadf0acbb744516027a0a59605c13d8f5c53848"} Nov 03 21:35:28.262574 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:28.262546 5041 generic.go:356] "Generic (PLEG): container finished" podID="411326e6-c41f-4ba7-b116-b909f9edd775" containerID="66f1cea16b547818887405f193bb90662d146d014c869ad929d37a55b86b6c29" exitCode=0 Nov 03 21:35:28.262724 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:28.262636 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" event={"ID":"411326e6-c41f-4ba7-b116-b909f9edd775","Type":"ContainerDied","Data":"66f1cea16b547818887405f193bb90662d146d014c869ad929d37a55b86b6c29"} Nov 03 21:35:28.264425 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:28.264399 5041 generic.go:356] "Generic (PLEG): container finished" podID="5f3f49aa-e369-46c3-8e35-5749f848114d" containerID="75d95d3c6d164daf5c0b1463b6db31a6a8a67052ae8d0a5047f3d9aa982d3c51" exitCode=0 Nov 03 21:35:28.264536 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:28.264510 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" event={"ID":"5f3f49aa-e369-46c3-8e35-5749f848114d","Type":"ContainerDied","Data":"75d95d3c6d164daf5c0b1463b6db31a6a8a67052ae8d0a5047f3d9aa982d3c51"} Nov 03 21:35:29.271008 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:29.270974 5041 generic.go:356] "Generic (PLEG): container finished" podID="a543bf82-126c-463e-9c78-d9353a1f69d8" containerID="8dd690342dba043e0eefb9f80057a7d9ddc8ccfcb8a0ed4d4a702c4e987ae8eb" exitCode=0 Nov 03 21:35:29.271409 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:29.271010 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" event={"ID":"a543bf82-126c-463e-9c78-d9353a1f69d8","Type":"ContainerDied","Data":"8dd690342dba043e0eefb9f80057a7d9ddc8ccfcb8a0ed4d4a702c4e987ae8eb"} Nov 03 21:35:29.273095 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:29.273068 5041 generic.go:356] "Generic (PLEG): container finished" podID="f25c1185-d4ca-4a23-b08c-0988faccc447" containerID="7f454a24e1083eb0b768056210b737a1ed37a62811244b9eb9fce21fd5f3a2f0" exitCode=0 Nov 03 21:35:29.273185 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:29.273111 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" event={"ID":"f25c1185-d4ca-4a23-b08c-0988faccc447","Type":"ContainerDied","Data":"7f454a24e1083eb0b768056210b737a1ed37a62811244b9eb9fce21fd5f3a2f0"} Nov 03 21:35:29.275409 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:29.275358 5041 generic.go:356] "Generic (PLEG): container finished" podID="8d360af4-8562-4b82-9c71-949a5183165b" containerID="72558eb107ffc349572c84b59446ecdd1abc0ef2a45a1af7c55d18a11b04fdf3" exitCode=0 Nov 03 21:35:29.275474 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:29.275447 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" event={"ID":"8d360af4-8562-4b82-9c71-949a5183165b","Type":"ContainerDied","Data":"72558eb107ffc349572c84b59446ecdd1abc0ef2a45a1af7c55d18a11b04fdf3"} Nov 03 21:35:29.277149 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:29.277117 5041 generic.go:356] "Generic (PLEG): container finished" podID="231f4e89-43ae-4acd-a012-405fc60841c6" containerID="3e827c80cd1f2b147a25db20373f76605eaa5c3626bece3fc319d1772b811d7c" exitCode=0 Nov 03 21:35:29.277261 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:29.277181 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" event={"ID":"231f4e89-43ae-4acd-a012-405fc60841c6","Type":"ContainerDied","Data":"3e827c80cd1f2b147a25db20373f76605eaa5c3626bece3fc319d1772b811d7c"} Nov 03 21:35:30.284173 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:30.284082 5041 generic.go:356] "Generic (PLEG): container finished" podID="411326e6-c41f-4ba7-b116-b909f9edd775" containerID="adbfd20a8b7212ebf716d3b0ae67d4ae65e1105ea046da13efd1c14fb294ee41" exitCode=0 Nov 03 21:35:30.284585 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:30.284171 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" event={"ID":"411326e6-c41f-4ba7-b116-b909f9edd775","Type":"ContainerDied","Data":"adbfd20a8b7212ebf716d3b0ae67d4ae65e1105ea046da13efd1c14fb294ee41"} Nov 03 21:35:30.287209 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:30.287182 5041 generic.go:356] "Generic (PLEG): container finished" podID="5f3f49aa-e369-46c3-8e35-5749f848114d" containerID="fe6208abca91dc303c759c3b735cff8857a2852df1b9bca9130d36f442814f21" exitCode=0 Nov 03 21:35:30.287305 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:30.287268 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" event={"ID":"5f3f49aa-e369-46c3-8e35-5749f848114d","Type":"ContainerDied","Data":"fe6208abca91dc303c759c3b735cff8857a2852df1b9bca9130d36f442814f21"} Nov 03 21:35:30.289759 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:30.289738 5041 generic.go:356] "Generic (PLEG): container finished" podID="a543bf82-126c-463e-9c78-d9353a1f69d8" containerID="092660d4a1aa5b84b4c038aecdac99f1824b9675cf3a8d91431bce41f0d88d1c" exitCode=0 Nov 03 21:35:30.289851 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:30.289823 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" event={"ID":"a543bf82-126c-463e-9c78-d9353a1f69d8","Type":"ContainerDied","Data":"092660d4a1aa5b84b4c038aecdac99f1824b9675cf3a8d91431bce41f0d88d1c"} Nov 03 21:35:30.292342 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:30.292298 5041 generic.go:356] "Generic (PLEG): container finished" podID="8d360af4-8562-4b82-9c71-949a5183165b" containerID="52b67a99bf9bc91f17f90a2250f69efb315ee5b1fdeae50e67d8b0953836db81" exitCode=0 Nov 03 21:35:30.292430 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:30.292401 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" event={"ID":"8d360af4-8562-4b82-9c71-949a5183165b","Type":"ContainerDied","Data":"52b67a99bf9bc91f17f90a2250f69efb315ee5b1fdeae50e67d8b0953836db81"} Nov 03 21:35:30.294883 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:30.294832 5041 generic.go:356] "Generic (PLEG): container finished" podID="231f4e89-43ae-4acd-a012-405fc60841c6" containerID="8452513ce42c0ac77bc75a4b85962d66f2a4295b330bc4ad892e65e5ec8f1836" exitCode=0 Nov 03 21:35:30.294970 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:30.294930 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" event={"ID":"231f4e89-43ae-4acd-a012-405fc60841c6","Type":"ContainerDied","Data":"8452513ce42c0ac77bc75a4b85962d66f2a4295b330bc4ad892e65e5ec8f1836"} Nov 03 21:35:30.297534 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:30.297513 5041 generic.go:356] "Generic (PLEG): container finished" podID="8e710705-b22a-470d-9cd0-15308632b744" containerID="dba5923a2fc76766d53bc2e7ef6400d7a111dd3ebb60b0efe4761b73bd1b89e6" exitCode=0 Nov 03 21:35:30.297616 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:30.297594 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" event={"ID":"8e710705-b22a-470d-9cd0-15308632b744","Type":"ContainerDied","Data":"dba5923a2fc76766d53bc2e7ef6400d7a111dd3ebb60b0efe4761b73bd1b89e6"} Nov 03 21:35:30.300886 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:30.300863 5041 generic.go:356] "Generic (PLEG): container finished" podID="9956d082-d0ee-459d-b604-1535b45a4e15" containerID="28ace98a731d9dbc7bc26be7e4817f0a7d8695d94f3ee26e858500680e804eb2" exitCode=0 Nov 03 21:35:30.300985 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:30.300914 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" event={"ID":"9956d082-d0ee-459d-b604-1535b45a4e15","Type":"ContainerDied","Data":"28ace98a731d9dbc7bc26be7e4817f0a7d8695d94f3ee26e858500680e804eb2"} Nov 03 21:35:37.413074 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:37.411514 5041 generic.go:356] "Generic (PLEG): container finished" podID="9956d082-d0ee-459d-b604-1535b45a4e15" containerID="0a1021417983f9efce06f7900d76d3548d3557514075b6e7959136e5d41b3a94" exitCode=0 Nov 03 21:35:37.413074 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:37.411621 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" event={"ID":"9956d082-d0ee-459d-b604-1535b45a4e15","Type":"ContainerDied","Data":"0a1021417983f9efce06f7900d76d3548d3557514075b6e7959136e5d41b3a94"} Nov 03 21:35:40.816470 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:35:40.816398 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:35:45.492431 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:45.492309 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" event={"ID":"f25c1185-d4ca-4a23-b08c-0988faccc447","Type":"ContainerStarted","Data":"b20f67167388d1f67d14374c0f50295fe0a18f2e68db7fcf1c3c95fe12abec04"} Nov 03 21:35:46.504616 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:46.504172 5041 generic.go:356] "Generic (PLEG): container finished" podID="f25c1185-d4ca-4a23-b08c-0988faccc447" containerID="b20f67167388d1f67d14374c0f50295fe0a18f2e68db7fcf1c3c95fe12abec04" exitCode=0 Nov 03 21:35:46.504616 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:46.504240 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" event={"ID":"f25c1185-d4ca-4a23-b08c-0988faccc447","Type":"ContainerDied","Data":"b20f67167388d1f67d14374c0f50295fe0a18f2e68db7fcf1c3c95fe12abec04"} Nov 03 21:35:47.511303 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:47.511268 5041 generic.go:356] "Generic (PLEG): container finished" podID="8e710705-b22a-470d-9cd0-15308632b744" containerID="b8f8549e753506036723ba56fc4d91fef3d3bda6de74746d20b16d734717912c" exitCode=0 Nov 03 21:35:47.511641 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:47.511346 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" event={"ID":"8e710705-b22a-470d-9cd0-15308632b744","Type":"ContainerDied","Data":"b8f8549e753506036723ba56fc4d91fef3d3bda6de74746d20b16d734717912c"} Nov 03 21:35:47.513419 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:47.513379 5041 generic.go:356] "Generic (PLEG): container finished" podID="411326e6-c41f-4ba7-b116-b909f9edd775" containerID="cefb7742eaff59a3df319f65bd25c052acb2c259366fbd5c589b8412ef9a3f69" exitCode=0 Nov 03 21:35:47.513501 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:47.513452 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" event={"ID":"411326e6-c41f-4ba7-b116-b909f9edd775","Type":"ContainerDied","Data":"cefb7742eaff59a3df319f65bd25c052acb2c259366fbd5c589b8412ef9a3f69"} Nov 03 21:35:47.653851 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:47.653802 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" Nov 03 21:35:47.739291 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:47.739261 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25c1185-d4ca-4a23-b08c-0988faccc447-kube-api-access-fbn67\" (UniqueName: \"kubernetes.io/projected/f25c1185-d4ca-4a23-b08c-0988faccc447-kube-api-access-fbn67\") pod \"f25c1185-d4ca-4a23-b08c-0988faccc447\" (UID: \"f25c1185-d4ca-4a23-b08c-0988faccc447\") " Nov 03 21:35:47.739437 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:47.739364 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f25c1185-d4ca-4a23-b08c-0988faccc447-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f25c1185-d4ca-4a23-b08c-0988faccc447-bundle\") pod \"f25c1185-d4ca-4a23-b08c-0988faccc447\" (UID: \"f25c1185-d4ca-4a23-b08c-0988faccc447\") " Nov 03 21:35:47.739437 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:47.739401 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f25c1185-d4ca-4a23-b08c-0988faccc447-util\" (UniqueName: \"kubernetes.io/empty-dir/f25c1185-d4ca-4a23-b08c-0988faccc447-util\") pod \"f25c1185-d4ca-4a23-b08c-0988faccc447\" (UID: \"f25c1185-d4ca-4a23-b08c-0988faccc447\") " Nov 03 21:35:47.740009 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:47.739972 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25c1185-d4ca-4a23-b08c-0988faccc447-bundle" pod "f25c1185-d4ca-4a23-b08c-0988faccc447" (UID: "f25c1185-d4ca-4a23-b08c-0988faccc447"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:47.741475 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:47.741445 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25c1185-d4ca-4a23-b08c-0988faccc447-kube-api-access-fbn67" pod "f25c1185-d4ca-4a23-b08c-0988faccc447" (UID: "f25c1185-d4ca-4a23-b08c-0988faccc447"). InnerVolumeSpecName "kube-api-access-fbn67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:47.745996 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:47.745969 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25c1185-d4ca-4a23-b08c-0988faccc447-util" pod "f25c1185-d4ca-4a23-b08c-0988faccc447" (UID: "f25c1185-d4ca-4a23-b08c-0988faccc447"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:47.840528 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:47.840495 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbn67\" (UniqueName: \"kubernetes.io/projected/f25c1185-d4ca-4a23-b08c-0988faccc447-kube-api-access-fbn67\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:47.840695 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:47.840531 5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f25c1185-d4ca-4a23-b08c-0988faccc447-bundle\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:47.840695 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:47.840549 5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f25c1185-d4ca-4a23-b08c-0988faccc447-util\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:48.520459 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.520428 5041 generic.go:356] "Generic (PLEG): container finished" podID="9956d082-d0ee-459d-b604-1535b45a4e15" containerID="7f258e4244878fab53b7d92fadd04501687167d83d89e7d4d86ed7cdce9fcc84" exitCode=0 Nov 03 21:35:48.520928 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.520524 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" event={"ID":"9956d082-d0ee-459d-b604-1535b45a4e15","Type":"ContainerDied","Data":"7f258e4244878fab53b7d92fadd04501687167d83d89e7d4d86ed7cdce9fcc84"} Nov 03 21:35:48.522496 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.522474 5041 generic.go:356] "Generic (PLEG): container finished" podID="5f3f49aa-e369-46c3-8e35-5749f848114d" containerID="ad186211cf21ce3ab9d13370aa138ddcfc24a44bb8535c9ff1dfe19456c15c3f" exitCode=0 Nov 03 21:35:48.522627 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.522562 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" event={"ID":"5f3f49aa-e369-46c3-8e35-5749f848114d","Type":"ContainerDied","Data":"ad186211cf21ce3ab9d13370aa138ddcfc24a44bb8535c9ff1dfe19456c15c3f"} Nov 03 21:35:48.524536 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.524508 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" event={"ID":"a543bf82-126c-463e-9c78-d9353a1f69d8","Type":"ContainerStarted","Data":"8acca2a104f1314c2a085ffe43db7f7520e5bc4c5d6ec7a9a6382831e1efe015"} Nov 03 21:35:48.526202 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.526184 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" event={"ID":"f25c1185-d4ca-4a23-b08c-0988faccc447","Type":"ContainerDied","Data":"b5ec2421e423d0b000f288a434c1dbfe541ef6381d5d3daa78c9172cc20cfe0d"} Nov 03 21:35:48.526292 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.526222 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5ec2421e423d0b000f288a434c1dbfe541ef6381d5d3daa78c9172cc20cfe0d" Nov 03 21:35:48.526292 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.526188 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aw9zmr" Nov 03 21:35:48.528410 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.528388 5041 generic.go:356] "Generic (PLEG): container finished" podID="8d360af4-8562-4b82-9c71-949a5183165b" containerID="46fb2d7394bf9da0fc1f27be6e0a58a8f3780cda424a2a803b4815a06b9f1f85" exitCode=0 Nov 03 21:35:48.528510 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.528468 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" event={"ID":"8d360af4-8562-4b82-9c71-949a5183165b","Type":"ContainerDied","Data":"46fb2d7394bf9da0fc1f27be6e0a58a8f3780cda424a2a803b4815a06b9f1f85"} Nov 03 21:35:48.530507 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.530484 5041 generic.go:356] "Generic (PLEG): container finished" podID="231f4e89-43ae-4acd-a012-405fc60841c6" containerID="d4fbe02bfde66236b4235c2151720b755c96fffaac826c6cdf87f48bb54687c2" exitCode=0 Nov 03 21:35:48.530610 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.530556 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" event={"ID":"231f4e89-43ae-4acd-a012-405fc60841c6","Type":"ContainerDied","Data":"d4fbe02bfde66236b4235c2151720b755c96fffaac826c6cdf87f48bb54687c2"} Nov 03 21:35:48.646185 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.646157 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" Nov 03 21:35:48.678963 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.678917 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" podStartSLOduration=2.314951339 podStartE2EDuration="30.678902246s" totalImagesPullingTime="28.363950907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:48.678138827 +0000 UTC m=+364.185367132" watchObservedRunningTime="2026-11-03 21:35:48.678902246 +0000 UTC m=+364.186130525" Nov 03 21:35:48.680870 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.680850 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" Nov 03 21:35:48.748267 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.748236 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/411326e6-c41f-4ba7-b116-b909f9edd775-bundle\" (UniqueName: \"kubernetes.io/empty-dir/411326e6-c41f-4ba7-b116-b909f9edd775-bundle\") pod \"411326e6-c41f-4ba7-b116-b909f9edd775\" (UID: \"411326e6-c41f-4ba7-b116-b909f9edd775\") " Nov 03 21:35:48.748398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.748276 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e710705-b22a-470d-9cd0-15308632b744-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e710705-b22a-470d-9cd0-15308632b744-bundle\") pod \"8e710705-b22a-470d-9cd0-15308632b744\" (UID: \"8e710705-b22a-470d-9cd0-15308632b744\") " Nov 03 21:35:48.748398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.748313 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/411326e6-c41f-4ba7-b116-b909f9edd775-util\" (UniqueName: \"kubernetes.io/empty-dir/411326e6-c41f-4ba7-b116-b909f9edd775-util\") pod \"411326e6-c41f-4ba7-b116-b909f9edd775\" (UID: \"411326e6-c41f-4ba7-b116-b909f9edd775\") " Nov 03 21:35:48.748398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.748332 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e710705-b22a-470d-9cd0-15308632b744-util\" (UniqueName: \"kubernetes.io/empty-dir/8e710705-b22a-470d-9cd0-15308632b744-util\") pod \"8e710705-b22a-470d-9cd0-15308632b744\" (UID: \"8e710705-b22a-470d-9cd0-15308632b744\") " Nov 03 21:35:48.748398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.748366 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/411326e6-c41f-4ba7-b116-b909f9edd775-kube-api-access-hxl8w\" (UniqueName: \"kubernetes.io/projected/411326e6-c41f-4ba7-b116-b909f9edd775-kube-api-access-hxl8w\") pod \"411326e6-c41f-4ba7-b116-b909f9edd775\" (UID: \"411326e6-c41f-4ba7-b116-b909f9edd775\") " Nov 03 21:35:48.748627 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.748421 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e710705-b22a-470d-9cd0-15308632b744-kube-api-access-5ssbm\" (UniqueName: \"kubernetes.io/projected/8e710705-b22a-470d-9cd0-15308632b744-kube-api-access-5ssbm\") pod \"8e710705-b22a-470d-9cd0-15308632b744\" (UID: \"8e710705-b22a-470d-9cd0-15308632b744\") " Nov 03 21:35:48.748909 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.748879 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411326e6-c41f-4ba7-b116-b909f9edd775-bundle" pod "411326e6-c41f-4ba7-b116-b909f9edd775" (UID: "411326e6-c41f-4ba7-b116-b909f9edd775"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:48.749180 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.749119 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e710705-b22a-470d-9cd0-15308632b744-bundle" pod "8e710705-b22a-470d-9cd0-15308632b744" (UID: "8e710705-b22a-470d-9cd0-15308632b744"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:48.751265 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.751237 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e710705-b22a-470d-9cd0-15308632b744-kube-api-access-5ssbm" pod "8e710705-b22a-470d-9cd0-15308632b744" (UID: "8e710705-b22a-470d-9cd0-15308632b744"). InnerVolumeSpecName "kube-api-access-5ssbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:48.751359 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.751253 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411326e6-c41f-4ba7-b116-b909f9edd775-kube-api-access-hxl8w" pod "411326e6-c41f-4ba7-b116-b909f9edd775" (UID: "411326e6-c41f-4ba7-b116-b909f9edd775"). InnerVolumeSpecName "kube-api-access-hxl8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:48.754195 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.754171 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/411326e6-c41f-4ba7-b116-b909f9edd775-util" pod "411326e6-c41f-4ba7-b116-b909f9edd775" (UID: "411326e6-c41f-4ba7-b116-b909f9edd775"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:48.754323 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.754304 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e710705-b22a-470d-9cd0-15308632b744-util" pod "8e710705-b22a-470d-9cd0-15308632b744" (UID: "8e710705-b22a-470d-9cd0-15308632b744"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:48.849286 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.849219 5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/411326e6-c41f-4ba7-b116-b909f9edd775-util\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:48.849286 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.849249 5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e710705-b22a-470d-9cd0-15308632b744-util\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:48.849286 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.849264 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxl8w\" (UniqueName: \"kubernetes.io/projected/411326e6-c41f-4ba7-b116-b909f9edd775-kube-api-access-hxl8w\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:48.849286 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.849278 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ssbm\" (UniqueName: \"kubernetes.io/projected/8e710705-b22a-470d-9cd0-15308632b744-kube-api-access-5ssbm\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:48.849529 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.849292 5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/411326e6-c41f-4ba7-b116-b909f9edd775-bundle\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:48.849529 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:48.849308 5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e710705-b22a-470d-9cd0-15308632b744-bundle\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:49.536555 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.536523 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" Nov 03 21:35:49.537053 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.536524 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fphrj" event={"ID":"8e710705-b22a-470d-9cd0-15308632b744","Type":"ContainerDied","Data":"94120c47ae2c625939dfd3701207a09ee60c2550717e3d348817ce5d84f347b8"} Nov 03 21:35:49.537053 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.536645 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94120c47ae2c625939dfd3701207a09ee60c2550717e3d348817ce5d84f347b8" Nov 03 21:35:49.538495 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.538477 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" Nov 03 21:35:49.538633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.538494 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4sv62" event={"ID":"411326e6-c41f-4ba7-b116-b909f9edd775","Type":"ContainerDied","Data":"f3a57f3910abed10242382500c82582bb029704f9e6f668fb96410019cb4c3df"} Nov 03 21:35:49.538633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.538530 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3a57f3910abed10242382500c82582bb029704f9e6f668fb96410019cb4c3df" Nov 03 21:35:49.540731 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.540710 5041 generic.go:356] "Generic (PLEG): container finished" podID="a543bf82-126c-463e-9c78-d9353a1f69d8" containerID="8acca2a104f1314c2a085ffe43db7f7520e5bc4c5d6ec7a9a6382831e1efe015" exitCode=0 Nov 03 21:35:49.540827 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.540787 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" event={"ID":"a543bf82-126c-463e-9c78-d9353a1f69d8","Type":"ContainerDied","Data":"8acca2a104f1314c2a085ffe43db7f7520e5bc4c5d6ec7a9a6382831e1efe015"} Nov 03 21:35:49.685299 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.685268 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" Nov 03 21:35:49.726286 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.726259 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" Nov 03 21:35:49.730888 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.730864 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" Nov 03 21:35:49.738367 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.738346 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" Nov 03 21:35:49.758143 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.758122 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231f4e89-43ae-4acd-a012-405fc60841c6-kube-api-access-sm4ph\" (UniqueName: \"kubernetes.io/projected/231f4e89-43ae-4acd-a012-405fc60841c6-kube-api-access-sm4ph\") pod \"231f4e89-43ae-4acd-a012-405fc60841c6\" (UID: \"231f4e89-43ae-4acd-a012-405fc60841c6\") " Nov 03 21:35:49.758270 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.758156 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231f4e89-43ae-4acd-a012-405fc60841c6-util\" (UniqueName: \"kubernetes.io/empty-dir/231f4e89-43ae-4acd-a012-405fc60841c6-util\") pod \"231f4e89-43ae-4acd-a012-405fc60841c6\" (UID: \"231f4e89-43ae-4acd-a012-405fc60841c6\") " Nov 03 21:35:49.758270 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.758186 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d360af4-8562-4b82-9c71-949a5183165b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d360af4-8562-4b82-9c71-949a5183165b-bundle\") pod \"8d360af4-8562-4b82-9c71-949a5183165b\" (UID: \"8d360af4-8562-4b82-9c71-949a5183165b\") " Nov 03 21:35:49.758435 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.758360 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3f49aa-e369-46c3-8e35-5749f848114d-util\" (UniqueName: \"kubernetes.io/empty-dir/5f3f49aa-e369-46c3-8e35-5749f848114d-util\") pod \"5f3f49aa-e369-46c3-8e35-5749f848114d\" (UID: \"5f3f49aa-e369-46c3-8e35-5749f848114d\") " Nov 03 21:35:49.758435 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.758395 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3f49aa-e369-46c3-8e35-5749f848114d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f3f49aa-e369-46c3-8e35-5749f848114d-bundle\") pod \"5f3f49aa-e369-46c3-8e35-5749f848114d\" (UID: \"5f3f49aa-e369-46c3-8e35-5749f848114d\") " Nov 03 21:35:49.758536 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.758435 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3f49aa-e369-46c3-8e35-5749f848114d-kube-api-access-cq97g\" (UniqueName: \"kubernetes.io/projected/5f3f49aa-e369-46c3-8e35-5749f848114d-kube-api-access-cq97g\") pod \"5f3f49aa-e369-46c3-8e35-5749f848114d\" (UID: \"5f3f49aa-e369-46c3-8e35-5749f848114d\") " Nov 03 21:35:49.758536 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.758460 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231f4e89-43ae-4acd-a012-405fc60841c6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/231f4e89-43ae-4acd-a012-405fc60841c6-bundle\") pod \"231f4e89-43ae-4acd-a012-405fc60841c6\" (UID: \"231f4e89-43ae-4acd-a012-405fc60841c6\") " Nov 03 21:35:49.758536 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.758508 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d360af4-8562-4b82-9c71-949a5183165b-kube-api-access-sth5k\" (UniqueName: \"kubernetes.io/projected/8d360af4-8562-4b82-9c71-949a5183165b-kube-api-access-sth5k\") pod \"8d360af4-8562-4b82-9c71-949a5183165b\" (UID: \"8d360af4-8562-4b82-9c71-949a5183165b\") " Nov 03 21:35:49.758716 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.758601 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d360af4-8562-4b82-9c71-949a5183165b-util\" (UniqueName: \"kubernetes.io/empty-dir/8d360af4-8562-4b82-9c71-949a5183165b-util\") pod \"8d360af4-8562-4b82-9c71-949a5183165b\" (UID: \"8d360af4-8562-4b82-9c71-949a5183165b\") " Nov 03 21:35:49.759016 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.758984 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3f49aa-e369-46c3-8e35-5749f848114d-bundle" pod "5f3f49aa-e369-46c3-8e35-5749f848114d" (UID: "5f3f49aa-e369-46c3-8e35-5749f848114d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:49.759407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.759372 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231f4e89-43ae-4acd-a012-405fc60841c6-bundle" pod "231f4e89-43ae-4acd-a012-405fc60841c6" (UID: "231f4e89-43ae-4acd-a012-405fc60841c6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:49.759786 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.759766 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d360af4-8562-4b82-9c71-949a5183165b-bundle" pod "8d360af4-8562-4b82-9c71-949a5183165b" (UID: "8d360af4-8562-4b82-9c71-949a5183165b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:49.760643 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.760616 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231f4e89-43ae-4acd-a012-405fc60841c6-kube-api-access-sm4ph" pod "231f4e89-43ae-4acd-a012-405fc60841c6" (UID: "231f4e89-43ae-4acd-a012-405fc60841c6"). InnerVolumeSpecName "kube-api-access-sm4ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:49.761088 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.761065 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d360af4-8562-4b82-9c71-949a5183165b-kube-api-access-sth5k" pod "8d360af4-8562-4b82-9c71-949a5183165b" (UID: "8d360af4-8562-4b82-9c71-949a5183165b"). InnerVolumeSpecName "kube-api-access-sth5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:49.761200 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.761182 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3f49aa-e369-46c3-8e35-5749f848114d-kube-api-access-cq97g" pod "5f3f49aa-e369-46c3-8e35-5749f848114d" (UID: "5f3f49aa-e369-46c3-8e35-5749f848114d"). InnerVolumeSpecName "kube-api-access-cq97g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:49.763211 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.763188 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3f49aa-e369-46c3-8e35-5749f848114d-util" pod "5f3f49aa-e369-46c3-8e35-5749f848114d" (UID: "5f3f49aa-e369-46c3-8e35-5749f848114d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:49.765106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.765083 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231f4e89-43ae-4acd-a012-405fc60841c6-util" pod "231f4e89-43ae-4acd-a012-405fc60841c6" (UID: "231f4e89-43ae-4acd-a012-405fc60841c6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:49.765241 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.765221 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d360af4-8562-4b82-9c71-949a5183165b-util" pod "8d360af4-8562-4b82-9c71-949a5183165b" (UID: "8d360af4-8562-4b82-9c71-949a5183165b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:49.860086 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.860013 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9956d082-d0ee-459d-b604-1535b45a4e15-util\" (UniqueName: \"kubernetes.io/empty-dir/9956d082-d0ee-459d-b604-1535b45a4e15-util\") pod \"9956d082-d0ee-459d-b604-1535b45a4e15\" (UID: \"9956d082-d0ee-459d-b604-1535b45a4e15\") " Nov 03 21:35:49.860086 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.860065 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9956d082-d0ee-459d-b604-1535b45a4e15-kube-api-access-g59vb\" (UniqueName: \"kubernetes.io/projected/9956d082-d0ee-459d-b604-1535b45a4e15-kube-api-access-g59vb\") pod \"9956d082-d0ee-459d-b604-1535b45a4e15\" (UID: \"9956d082-d0ee-459d-b604-1535b45a4e15\") " Nov 03 21:35:49.860258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.860105 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9956d082-d0ee-459d-b604-1535b45a4e15-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9956d082-d0ee-459d-b604-1535b45a4e15-bundle\") pod \"9956d082-d0ee-459d-b604-1535b45a4e15\" (UID: \"9956d082-d0ee-459d-b604-1535b45a4e15\") " Nov 03 21:35:49.860258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.860247 5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d360af4-8562-4b82-9c71-949a5183165b-util\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:49.860340 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.860267 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm4ph\" (UniqueName: \"kubernetes.io/projected/231f4e89-43ae-4acd-a012-405fc60841c6-kube-api-access-sm4ph\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:49.860340 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.860281 5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/231f4e89-43ae-4acd-a012-405fc60841c6-util\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:49.860340 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.860296 5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d360af4-8562-4b82-9c71-949a5183165b-bundle\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:49.860340 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.860309 5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f3f49aa-e369-46c3-8e35-5749f848114d-util\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:49.860340 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.860321 5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f3f49aa-e369-46c3-8e35-5749f848114d-bundle\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:49.860497 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.860352 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq97g\" (UniqueName: \"kubernetes.io/projected/5f3f49aa-e369-46c3-8e35-5749f848114d-kube-api-access-cq97g\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:49.860497 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.860366 5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/231f4e89-43ae-4acd-a012-405fc60841c6-bundle\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:49.860497 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.860379 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sth5k\" (UniqueName: \"kubernetes.io/projected/8d360af4-8562-4b82-9c71-949a5183165b-kube-api-access-sth5k\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:49.860600 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.860556 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9956d082-d0ee-459d-b604-1535b45a4e15-bundle" pod "9956d082-d0ee-459d-b604-1535b45a4e15" (UID: "9956d082-d0ee-459d-b604-1535b45a4e15"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:49.862171 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.862144 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9956d082-d0ee-459d-b604-1535b45a4e15-kube-api-access-g59vb" pod "9956d082-d0ee-459d-b604-1535b45a4e15" (UID: "9956d082-d0ee-459d-b604-1535b45a4e15"). InnerVolumeSpecName "kube-api-access-g59vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:49.865542 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.865521 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9956d082-d0ee-459d-b604-1535b45a4e15-util" pod "9956d082-d0ee-459d-b604-1535b45a4e15" (UID: "9956d082-d0ee-459d-b604-1535b45a4e15"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:49.961323 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.961298 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g59vb\" (UniqueName: \"kubernetes.io/projected/9956d082-d0ee-459d-b604-1535b45a4e15-kube-api-access-g59vb\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:49.961323 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.961323 5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9956d082-d0ee-459d-b604-1535b45a4e15-bundle\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:49.961435 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:49.961332 5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9956d082-d0ee-459d-b604-1535b45a4e15-util\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:50.546650 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.546611 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" event={"ID":"8d360af4-8562-4b82-9c71-949a5183165b","Type":"ContainerDied","Data":"f1f8d331d1f929f5d102e7099e684bd99012fe476e87787ba713f471eb8201f1"} Nov 03 21:35:50.546650 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.546624 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c224wq" Nov 03 21:35:50.547163 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.546707 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1f8d331d1f929f5d102e7099e684bd99012fe476e87787ba713f471eb8201f1" Nov 03 21:35:50.548536 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.548506 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" Nov 03 21:35:50.548536 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.548510 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71p9xsm" event={"ID":"231f4e89-43ae-4acd-a012-405fc60841c6","Type":"ContainerDied","Data":"d696b010a2f9e18fd8891d2a143636b48720e26d46c0651a745332e93c50afd4"} Nov 03 21:35:50.548748 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.548570 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d696b010a2f9e18fd8891d2a143636b48720e26d46c0651a745332e93c50afd4" Nov 03 21:35:50.550459 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.550437 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" Nov 03 21:35:50.550562 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.550457 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644mvkcv" event={"ID":"9956d082-d0ee-459d-b604-1535b45a4e15","Type":"ContainerDied","Data":"f743f5dcf5d6ed197cda9b8b045e71dbdfc71ff9e3d5606f5b82b38a14ea1f64"} Nov 03 21:35:50.550562 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.550492 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f743f5dcf5d6ed197cda9b8b045e71dbdfc71ff9e3d5606f5b82b38a14ea1f64" Nov 03 21:35:50.552316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.552296 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" Nov 03 21:35:50.552409 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.552323 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74slw9" event={"ID":"5f3f49aa-e369-46c3-8e35-5749f848114d","Type":"ContainerDied","Data":"5375153aae4a2157d50783cb9bf9d1eb8cbf1ab30d457308220125b12e3c98fc"} Nov 03 21:35:50.552409 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.552358 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5375153aae4a2157d50783cb9bf9d1eb8cbf1ab30d457308220125b12e3c98fc" Nov 03 21:35:50.683706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.683684 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" Nov 03 21:35:50.768595 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.768564 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a543bf82-126c-463e-9c78-d9353a1f69d8-kube-api-access-qqwmp\" (UniqueName: \"kubernetes.io/projected/a543bf82-126c-463e-9c78-d9353a1f69d8-kube-api-access-qqwmp\") pod \"a543bf82-126c-463e-9c78-d9353a1f69d8\" (UID: \"a543bf82-126c-463e-9c78-d9353a1f69d8\") " Nov 03 21:35:50.768740 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.768623 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a543bf82-126c-463e-9c78-d9353a1f69d8-util\" (UniqueName: \"kubernetes.io/empty-dir/a543bf82-126c-463e-9c78-d9353a1f69d8-util\") pod \"a543bf82-126c-463e-9c78-d9353a1f69d8\" (UID: \"a543bf82-126c-463e-9c78-d9353a1f69d8\") " Nov 03 21:35:50.768740 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.768683 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a543bf82-126c-463e-9c78-d9353a1f69d8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a543bf82-126c-463e-9c78-d9353a1f69d8-bundle\") pod \"a543bf82-126c-463e-9c78-d9353a1f69d8\" (UID: \"a543bf82-126c-463e-9c78-d9353a1f69d8\") " Nov 03 21:35:50.769155 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.769126 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a543bf82-126c-463e-9c78-d9353a1f69d8-bundle" pod "a543bf82-126c-463e-9c78-d9353a1f69d8" (UID: "a543bf82-126c-463e-9c78-d9353a1f69d8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:50.770451 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.770425 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a543bf82-126c-463e-9c78-d9353a1f69d8-kube-api-access-qqwmp" pod "a543bf82-126c-463e-9c78-d9353a1f69d8" (UID: "a543bf82-126c-463e-9c78-d9353a1f69d8"). InnerVolumeSpecName "kube-api-access-qqwmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:50.773796 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.773774 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a543bf82-126c-463e-9c78-d9353a1f69d8-util" pod "a543bf82-126c-463e-9c78-d9353a1f69d8" (UID: "a543bf82-126c-463e-9c78-d9353a1f69d8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:35:50.869403 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.869327 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqwmp\" (UniqueName: \"kubernetes.io/projected/a543bf82-126c-463e-9c78-d9353a1f69d8-kube-api-access-qqwmp\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:50.869403 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.869355 5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a543bf82-126c-463e-9c78-d9353a1f69d8-util\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:50.869403 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:50.869369 5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a543bf82-126c-463e-9c78-d9353a1f69d8-bundle\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:35:51.558105 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:51.558078 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" Nov 03 21:35:51.558105 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:51.558093 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d779mv6" event={"ID":"a543bf82-126c-463e-9c78-d9353a1f69d8","Type":"ContainerDied","Data":"fc442313f4917e71d9f590a55296721658a43301aad77dc2642a8f212f414548"} Nov 03 21:35:51.558556 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:51.558137 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc442313f4917e71d9f590a55296721658a43301aad77dc2642a8f212f414548" Nov 03 21:35:58.772177 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772144 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-bpbtm"] Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772452 5041 cpu_manager.go:414] "containerMap: removing container" podUID="411326e6-c41f-4ba7-b116-b909f9edd775" containerName="extract" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772463 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="411326e6-c41f-4ba7-b116-b909f9edd775" containerName="extract" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772475 5041 cpu_manager.go:414] "containerMap: removing container" podUID="9956d082-d0ee-459d-b604-1535b45a4e15" containerName="pull" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772481 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9956d082-d0ee-459d-b604-1535b45a4e15" containerName="pull" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772489 5041 cpu_manager.go:414] "containerMap: removing container" podUID="9956d082-d0ee-459d-b604-1535b45a4e15" containerName="extract" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772494 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9956d082-d0ee-459d-b604-1535b45a4e15" containerName="extract" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772500 5041 cpu_manager.go:414] "containerMap: removing container" podUID="f25c1185-d4ca-4a23-b08c-0988faccc447" containerName="pull" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772504 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25c1185-d4ca-4a23-b08c-0988faccc447" containerName="pull" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772510 5041 cpu_manager.go:414] "containerMap: removing container" podUID="8e710705-b22a-470d-9cd0-15308632b744" containerName="extract" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772514 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e710705-b22a-470d-9cd0-15308632b744" containerName="extract" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772521 5041 cpu_manager.go:414] "containerMap: removing container" podUID="8e710705-b22a-470d-9cd0-15308632b744" containerName="util" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772532 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e710705-b22a-470d-9cd0-15308632b744" containerName="util" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772540 5041 cpu_manager.go:414] "containerMap: removing container" podUID="5f3f49aa-e369-46c3-8e35-5749f848114d" containerName="util" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772544 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3f49aa-e369-46c3-8e35-5749f848114d" containerName="util" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772548 5041 cpu_manager.go:414] "containerMap: removing container" podUID="f25c1185-d4ca-4a23-b08c-0988faccc447" containerName="util" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772553 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25c1185-d4ca-4a23-b08c-0988faccc447" containerName="util" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772562 5041 cpu_manager.go:414] "containerMap: removing container" podUID="231f4e89-43ae-4acd-a012-405fc60841c6" containerName="pull" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772567 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231f4e89-43ae-4acd-a012-405fc60841c6" containerName="pull" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772577 5041 cpu_manager.go:414] "containerMap: removing container" podUID="411326e6-c41f-4ba7-b116-b909f9edd775" containerName="util" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772581 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="411326e6-c41f-4ba7-b116-b909f9edd775" containerName="util" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772588 5041 cpu_manager.go:414] "containerMap: removing container" podUID="a543bf82-126c-463e-9c78-d9353a1f69d8" containerName="util" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772592 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a543bf82-126c-463e-9c78-d9353a1f69d8" containerName="util" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772603 5041 cpu_manager.go:414] "containerMap: removing container" podUID="231f4e89-43ae-4acd-a012-405fc60841c6" containerName="util" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772607 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231f4e89-43ae-4acd-a012-405fc60841c6" containerName="util" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772612 5041 cpu_manager.go:414] "containerMap: removing container" podUID="a543bf82-126c-463e-9c78-d9353a1f69d8" containerName="pull" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772618 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a543bf82-126c-463e-9c78-d9353a1f69d8" containerName="pull" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772625 5041 cpu_manager.go:414] "containerMap: removing container" podUID="8e710705-b22a-470d-9cd0-15308632b744" containerName="pull" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772629 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e710705-b22a-470d-9cd0-15308632b744" containerName="pull" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772637 5041 cpu_manager.go:414] "containerMap: removing container" podUID="8d360af4-8562-4b82-9c71-949a5183165b" containerName="util" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772642 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d360af4-8562-4b82-9c71-949a5183165b" containerName="util" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772650 5041 cpu_manager.go:414] "containerMap: removing container" podUID="f25c1185-d4ca-4a23-b08c-0988faccc447" containerName="extract" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772655 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25c1185-d4ca-4a23-b08c-0988faccc447" containerName="extract" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772677 5041 cpu_manager.go:414] "containerMap: removing container" podUID="5f3f49aa-e369-46c3-8e35-5749f848114d" containerName="pull" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772684 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3f49aa-e369-46c3-8e35-5749f848114d" containerName="pull" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772691 5041 cpu_manager.go:414] "containerMap: removing container" podUID="8d360af4-8562-4b82-9c71-949a5183165b" containerName="extract" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772696 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d360af4-8562-4b82-9c71-949a5183165b" containerName="extract" Nov 03 21:35:58.772684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772703 5041 cpu_manager.go:414] "containerMap: removing container" podUID="5f3f49aa-e369-46c3-8e35-5749f848114d" containerName="extract" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772708 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3f49aa-e369-46c3-8e35-5749f848114d" containerName="extract" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772716 5041 cpu_manager.go:414] "containerMap: removing container" podUID="411326e6-c41f-4ba7-b116-b909f9edd775" containerName="pull" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772720 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="411326e6-c41f-4ba7-b116-b909f9edd775" containerName="pull" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772726 5041 cpu_manager.go:414] "containerMap: removing container" podUID="8d360af4-8562-4b82-9c71-949a5183165b" containerName="pull" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772730 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d360af4-8562-4b82-9c71-949a5183165b" containerName="pull" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772737 5041 cpu_manager.go:414] "containerMap: removing container" podUID="9956d082-d0ee-459d-b604-1535b45a4e15" containerName="util" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772741 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9956d082-d0ee-459d-b604-1535b45a4e15" containerName="util" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772749 5041 cpu_manager.go:414] "containerMap: removing container" podUID="231f4e89-43ae-4acd-a012-405fc60841c6" containerName="extract" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772755 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231f4e89-43ae-4acd-a012-405fc60841c6" containerName="extract" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772764 5041 cpu_manager.go:414] "containerMap: removing container" podUID="a543bf82-126c-463e-9c78-d9353a1f69d8" containerName="extract" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772770 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a543bf82-126c-463e-9c78-d9353a1f69d8" containerName="extract" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772836 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25c1185-d4ca-4a23-b08c-0988faccc447" containerName="extract" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772847 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d360af4-8562-4b82-9c71-949a5183165b" containerName="extract" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772855 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3f49aa-e369-46c3-8e35-5749f848114d" containerName="extract" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772864 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="231f4e89-43ae-4acd-a012-405fc60841c6" containerName="extract" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772872 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a543bf82-126c-463e-9c78-d9353a1f69d8" containerName="extract" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772883 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="411326e6-c41f-4ba7-b116-b909f9edd775" containerName="extract" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772891 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9956d082-d0ee-459d-b604-1535b45a4e15" containerName="extract" Nov 03 21:35:58.774392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.772896 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e710705-b22a-470d-9cd0-15308632b744" containerName="extract" Nov 03 21:35:58.775759 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.775739 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-bpbtm" Nov 03 21:35:58.793936 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.793909 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-bpbtm"] Nov 03 21:35:58.805935 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.805910 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 21:35:58.806065 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.805985 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-dwsgg\"" Nov 03 21:35:58.806065 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.806013 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 21:35:58.835275 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.835247 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkrdm\" (UniqueName: \"kubernetes.io/projected/49f22d2e-abcf-4352-89bf-e534d86e3cb0-kube-api-access-xkrdm\") pod \"cert-manager-597b95cb84-bpbtm\" (UID: \"49f22d2e-abcf-4352-89bf-e534d86e3cb0\") " pod="cert-manager/cert-manager-597b95cb84-bpbtm" Nov 03 21:35:58.835421 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.835283 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49f22d2e-abcf-4352-89bf-e534d86e3cb0-bound-sa-token\") pod \"cert-manager-597b95cb84-bpbtm\" (UID: \"49f22d2e-abcf-4352-89bf-e534d86e3cb0\") " pod="cert-manager/cert-manager-597b95cb84-bpbtm" Nov 03 21:35:58.936111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.936062 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkrdm\" (UniqueName: \"kubernetes.io/projected/49f22d2e-abcf-4352-89bf-e534d86e3cb0-kube-api-access-xkrdm\") pod \"cert-manager-597b95cb84-bpbtm\" (UID: \"49f22d2e-abcf-4352-89bf-e534d86e3cb0\") " pod="cert-manager/cert-manager-597b95cb84-bpbtm" Nov 03 21:35:58.936285 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.936132 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49f22d2e-abcf-4352-89bf-e534d86e3cb0-bound-sa-token\") pod \"cert-manager-597b95cb84-bpbtm\" (UID: \"49f22d2e-abcf-4352-89bf-e534d86e3cb0\") " pod="cert-manager/cert-manager-597b95cb84-bpbtm" Nov 03 21:35:58.956712 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.956681 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49f22d2e-abcf-4352-89bf-e534d86e3cb0-bound-sa-token\") pod \"cert-manager-597b95cb84-bpbtm\" (UID: \"49f22d2e-abcf-4352-89bf-e534d86e3cb0\") " pod="cert-manager/cert-manager-597b95cb84-bpbtm" Nov 03 21:35:58.956853 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:58.956824 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkrdm\" (UniqueName: \"kubernetes.io/projected/49f22d2e-abcf-4352-89bf-e534d86e3cb0-kube-api-access-xkrdm\") pod \"cert-manager-597b95cb84-bpbtm\" (UID: \"49f22d2e-abcf-4352-89bf-e534d86e3cb0\") " pod="cert-manager/cert-manager-597b95cb84-bpbtm" Nov 03 21:35:59.100931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:59.100844 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-bpbtm" Nov 03 21:35:59.392790 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:59.392710 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-bpbtm"] Nov 03 21:35:59.395769 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:35:59.395735 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49f22d2e_abcf_4352_89bf_e534d86e3cb0.slice/crio-1ed93e5fca822b9e222c3a6a02a78852d0861027decbf64b2ef41f9afbe5324f WatchSource:0}: Error finding container 1ed93e5fca822b9e222c3a6a02a78852d0861027decbf64b2ef41f9afbe5324f: Status 404 returned error can't find the container with id 1ed93e5fca822b9e222c3a6a02a78852d0861027decbf64b2ef41f9afbe5324f Nov 03 21:35:59.595292 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:35:59.595255 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-bpbtm" event={"ID":"49f22d2e-abcf-4352-89bf-e534d86e3cb0","Type":"ContainerStarted","Data":"1ed93e5fca822b9e222c3a6a02a78852d0861027decbf64b2ef41f9afbe5324f"} Nov 03 21:36:02.611641 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:02.611609 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-bpbtm" event={"ID":"49f22d2e-abcf-4352-89bf-e534d86e3cb0","Type":"ContainerStarted","Data":"168c156d0aa120b62113de24b9e3fea5a135e7a85c1dbedbcced904ae1e29506"} Nov 03 21:36:02.652807 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:02.652762 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-bpbtm" podStartSLOduration=1.6331554000000001 podStartE2EDuration="4.652744757s" totalImagesPullingTime="3.019589357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:02.65179837 +0000 UTC m=+378.159026665" watchObservedRunningTime="2026-11-03 21:36:02.652744757 +0000 UTC m=+378.159973037" Nov 03 21:36:04.227088 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.227054 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn"] Nov 03 21:36:04.229601 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.229584 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.260122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.260083 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 21:36:04.260122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.260104 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 21:36:04.260302 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.260126 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 21:36:04.260302 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.260142 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 21:36:04.260302 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.260185 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-zrtwp\"" Nov 03 21:36:04.260302 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.260242 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 21:36:04.272156 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.272121 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn"] Nov 03 21:36:04.389854 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.389827 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a04cf214-5b6c-4c5e-b59e-a9912c743f56-tmp\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.390023 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.389864 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a04cf214-5b6c-4c5e-b59e-a9912c743f56-webhook-cert\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.390023 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.389885 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a04cf214-5b6c-4c5e-b59e-a9912c743f56-metrics-cert\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.390023 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.389995 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a04cf214-5b6c-4c5e-b59e-a9912c743f56-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.390023 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.390015 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a04cf214-5b6c-4c5e-b59e-a9912c743f56-manifests\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.390181 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.390032 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhjls\" (UniqueName: \"kubernetes.io/projected/a04cf214-5b6c-4c5e-b59e-a9912c743f56-kube-api-access-nhjls\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.404149 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.404117 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v"] Nov 03 21:36:04.406613 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.406594 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" Nov 03 21:36:04.429930 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.429904 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v"] Nov 03 21:36:04.447193 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.447173 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kprcl\"" Nov 03 21:36:04.447304 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.447233 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 21:36:04.447304 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.447287 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 21:36:04.490894 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.490844 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a04cf214-5b6c-4c5e-b59e-a9912c743f56-webhook-cert\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.490894 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.490877 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrcfm\" (UniqueName: \"kubernetes.io/projected/e2ea4359-b4cf-4827-9b65-86fa2cc39e60-kube-api-access-rrcfm\") pod \"kuadrant-operator-controller-manager-cf8469d7b-ptj5v\" (UID: \"e2ea4359-b4cf-4827-9b65-86fa2cc39e60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" Nov 03 21:36:04.491066 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.490903 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a04cf214-5b6c-4c5e-b59e-a9912c743f56-metrics-cert\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.491066 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.491012 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a04cf214-5b6c-4c5e-b59e-a9912c743f56-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.491066 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.491052 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a04cf214-5b6c-4c5e-b59e-a9912c743f56-manifests\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.491237 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.491083 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjls\" (UniqueName: \"kubernetes.io/projected/a04cf214-5b6c-4c5e-b59e-a9912c743f56-kube-api-access-nhjls\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.491237 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.491140 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2ea4359-b4cf-4827-9b65-86fa2cc39e60-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-ptj5v\" (UID: \"e2ea4359-b4cf-4827-9b65-86fa2cc39e60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" Nov 03 21:36:04.491237 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.491202 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a04cf214-5b6c-4c5e-b59e-a9912c743f56-tmp\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.491490 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.491461 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a04cf214-5b6c-4c5e-b59e-a9912c743f56-manifests\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.493310 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.493278 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a04cf214-5b6c-4c5e-b59e-a9912c743f56-tmp\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.493418 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.493408 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a04cf214-5b6c-4c5e-b59e-a9912c743f56-metrics-cert\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.493556 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.493538 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a04cf214-5b6c-4c5e-b59e-a9912c743f56-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.493611 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.493555 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a04cf214-5b6c-4c5e-b59e-a9912c743f56-webhook-cert\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.512893 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.512867 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhjls\" (UniqueName: \"kubernetes.io/projected/a04cf214-5b6c-4c5e-b59e-a9912c743f56-kube-api-access-nhjls\") pod \"opendatahub-operator-controller-manager-5754f75c9f-md9nn\" (UID: \"a04cf214-5b6c-4c5e-b59e-a9912c743f56\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.543147 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.543123 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:04.592941 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.592264 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrcfm\" (UniqueName: \"kubernetes.io/projected/e2ea4359-b4cf-4827-9b65-86fa2cc39e60-kube-api-access-rrcfm\") pod \"kuadrant-operator-controller-manager-cf8469d7b-ptj5v\" (UID: \"e2ea4359-b4cf-4827-9b65-86fa2cc39e60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" Nov 03 21:36:04.592941 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.592407 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2ea4359-b4cf-4827-9b65-86fa2cc39e60-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-ptj5v\" (UID: \"e2ea4359-b4cf-4827-9b65-86fa2cc39e60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" Nov 03 21:36:04.592941 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.592868 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2ea4359-b4cf-4827-9b65-86fa2cc39e60-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-ptj5v\" (UID: \"e2ea4359-b4cf-4827-9b65-86fa2cc39e60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" Nov 03 21:36:04.614291 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.614266 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrcfm\" (UniqueName: \"kubernetes.io/projected/e2ea4359-b4cf-4827-9b65-86fa2cc39e60-kube-api-access-rrcfm\") pod \"kuadrant-operator-controller-manager-cf8469d7b-ptj5v\" (UID: \"e2ea4359-b4cf-4827-9b65-86fa2cc39e60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" Nov 03 21:36:04.717578 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.717544 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" Nov 03 21:36:04.886316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:04.886288 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn"] Nov 03 21:36:04.887802 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:36:04.887776 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04cf214_5b6c_4c5e_b59e_a9912c743f56.slice/crio-c7f7958fc1f84a6af87b7a55305b6f093e41cf8acca40e14401c5a8257db4649 WatchSource:0}: Error finding container c7f7958fc1f84a6af87b7a55305b6f093e41cf8acca40e14401c5a8257db4649: Status 404 returned error can't find the container with id c7f7958fc1f84a6af87b7a55305b6f093e41cf8acca40e14401c5a8257db4649 Nov 03 21:36:05.019564 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:05.019476 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v"] Nov 03 21:36:05.022545 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:36:05.022519 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ea4359_b4cf_4827_9b65_86fa2cc39e60.slice/crio-eb49609f8f7dd97fd686b21a3f4deea0b4ffa70d58a2002cda4a9df60e6eef03 WatchSource:0}: Error finding container eb49609f8f7dd97fd686b21a3f4deea0b4ffa70d58a2002cda4a9df60e6eef03: Status 404 returned error can't find the container with id eb49609f8f7dd97fd686b21a3f4deea0b4ffa70d58a2002cda4a9df60e6eef03 Nov 03 21:36:05.631842 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:05.631802 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" event={"ID":"a04cf214-5b6c-4c5e-b59e-a9912c743f56","Type":"ContainerStarted","Data":"c7f7958fc1f84a6af87b7a55305b6f093e41cf8acca40e14401c5a8257db4649"} Nov 03 21:36:05.634547 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:05.634512 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" event={"ID":"e2ea4359-b4cf-4827-9b65-86fa2cc39e60","Type":"ContainerStarted","Data":"eb49609f8f7dd97fd686b21a3f4deea0b4ffa70d58a2002cda4a9df60e6eef03"} Nov 03 21:36:07.655456 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:07.653870 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ghgpz"] Nov 03 21:36:07.660212 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:07.660185 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ghgpz" Nov 03 21:36:07.678612 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:07.678583 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dzffs\"" Nov 03 21:36:07.695391 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:07.695358 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ghgpz"] Nov 03 21:36:07.822880 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:07.822840 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2c2b\" (UniqueName: \"kubernetes.io/projected/418620a8-7e15-42ac-97fa-ad9568954b5f-kube-api-access-m2c2b\") pod \"limitador-operator-controller-manager-5778bb4449-ghgpz\" (UID: \"418620a8-7e15-42ac-97fa-ad9568954b5f\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ghgpz" Nov 03 21:36:07.924173 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:07.924091 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2c2b\" (UniqueName: \"kubernetes.io/projected/418620a8-7e15-42ac-97fa-ad9568954b5f-kube-api-access-m2c2b\") pod \"limitador-operator-controller-manager-5778bb4449-ghgpz\" (UID: \"418620a8-7e15-42ac-97fa-ad9568954b5f\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ghgpz" Nov 03 21:36:07.944957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:07.944925 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2c2b\" (UniqueName: \"kubernetes.io/projected/418620a8-7e15-42ac-97fa-ad9568954b5f-kube-api-access-m2c2b\") pod \"limitador-operator-controller-manager-5778bb4449-ghgpz\" (UID: \"418620a8-7e15-42ac-97fa-ad9568954b5f\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ghgpz" Nov 03 21:36:07.976326 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:07.976298 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ghgpz" Nov 03 21:36:08.287014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:08.286979 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ghgpz"] Nov 03 21:36:08.290210 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:36:08.290180 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418620a8_7e15_42ac_97fa_ad9568954b5f.slice/crio-fa5245408badf2b095ea253382d6e2124a81739fc32ea13ae3a1ea8e0d6cef51 WatchSource:0}: Error finding container fa5245408badf2b095ea253382d6e2124a81739fc32ea13ae3a1ea8e0d6cef51: Status 404 returned error can't find the container with id fa5245408badf2b095ea253382d6e2124a81739fc32ea13ae3a1ea8e0d6cef51 Nov 03 21:36:08.653245 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:08.653211 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ghgpz" event={"ID":"418620a8-7e15-42ac-97fa-ad9568954b5f","Type":"ContainerStarted","Data":"fa5245408badf2b095ea253382d6e2124a81739fc32ea13ae3a1ea8e0d6cef51"} Nov 03 21:36:08.655007 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:08.654970 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" event={"ID":"a04cf214-5b6c-4c5e-b59e-a9912c743f56","Type":"ContainerStarted","Data":"af31ad26c787b4d0d6d564f8a1f3146df194fd5c77037e08599a6062a9627b89"} Nov 03 21:36:09.662518 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:09.662478 5041 generic.go:356] "Generic (PLEG): container finished" podID="a04cf214-5b6c-4c5e-b59e-a9912c743f56" containerID="af31ad26c787b4d0d6d564f8a1f3146df194fd5c77037e08599a6062a9627b89" exitCode=0 Nov 03 21:36:09.663055 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:09.662545 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" event={"ID":"a04cf214-5b6c-4c5e-b59e-a9912c743f56","Type":"ContainerDied","Data":"af31ad26c787b4d0d6d564f8a1f3146df194fd5c77037e08599a6062a9627b89"} Nov 03 21:36:10.669843 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:10.669802 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" event={"ID":"a04cf214-5b6c-4c5e-b59e-a9912c743f56","Type":"ContainerStarted","Data":"c00adc0dcd8d40e515fc73bc063e92ef51befc4d822203a426e2a10ea9cd4793"} Nov 03 21:36:10.670252 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:10.669930 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:10.671540 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:10.671517 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" event={"ID":"e2ea4359-b4cf-4827-9b65-86fa2cc39e60","Type":"ContainerStarted","Data":"db2db5705800cb4fa4da36b187eedc4de9ba9f854c794e8fd11e761bc56e0418"} Nov 03 21:36:10.671695 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:10.671585 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" Nov 03 21:36:10.673043 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:10.673021 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ghgpz" event={"ID":"418620a8-7e15-42ac-97fa-ad9568954b5f","Type":"ContainerStarted","Data":"94259396168b8cac5119cb82aec459293503051f68bc8957cbac486961b20a8b"} Nov 03 21:36:10.673158 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:10.673132 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ghgpz" Nov 03 21:36:10.751653 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:10.751444 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" podStartSLOduration=2.391227708 podStartE2EDuration="6.751424672s" totalImagesPullingTime="3.742966s" totalInitContainerRuntime="617.230964ms" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:10.73700184 +0000 UTC m=+386.244230121" watchObservedRunningTime="2026-11-03 21:36:10.751424672 +0000 UTC m=+386.258652953" Nov 03 21:36:10.777287 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:36:10.777252 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:36:10.786754 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:10.786709 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" podStartSLOduration=1.319946665 podStartE2EDuration="6.786695595s" totalImagesPullingTime="5.46674893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:10.785369718 +0000 UTC m=+386.292597998" watchObservedRunningTime="2026-11-03 21:36:10.786695595 +0000 UTC m=+386.293923853" Nov 03 21:36:10.827064 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:10.827008 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ghgpz" podStartSLOduration=1.873185568 podStartE2EDuration="3.826990591s" totalImagesPullingTime="1.953805023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:10.825579641 +0000 UTC m=+386.332807921" watchObservedRunningTime="2026-11-03 21:36:10.826990591 +0000 UTC m=+386.334218874" Nov 03 21:36:12.943160 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:12.943124 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-c446k"] Nov 03 21:36:12.945817 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:12.945796 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c446k" Nov 03 21:36:12.969818 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:12.969784 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-c446k"] Nov 03 21:36:13.016637 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:13.016613 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-66zzw\"" Nov 03 21:36:13.071343 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:13.071309 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6vpd\" (UniqueName: \"kubernetes.io/projected/b3c86a2a-007c-4a88-87cc-5a97de621e31-kube-api-access-b6vpd\") pod \"authorino-operator-7b7bb4bc77-c446k\" (UID: \"b3c86a2a-007c-4a88-87cc-5a97de621e31\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-c446k" Nov 03 21:36:13.172367 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:13.172331 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6vpd\" (UniqueName: \"kubernetes.io/projected/b3c86a2a-007c-4a88-87cc-5a97de621e31-kube-api-access-b6vpd\") pod \"authorino-operator-7b7bb4bc77-c446k\" (UID: \"b3c86a2a-007c-4a88-87cc-5a97de621e31\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-c446k" Nov 03 21:36:13.194056 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:13.193991 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6vpd\" (UniqueName: \"kubernetes.io/projected/b3c86a2a-007c-4a88-87cc-5a97de621e31-kube-api-access-b6vpd\") pod \"authorino-operator-7b7bb4bc77-c446k\" (UID: \"b3c86a2a-007c-4a88-87cc-5a97de621e31\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-c446k" Nov 03 21:36:13.257837 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:13.257806 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c446k" Nov 03 21:36:13.551090 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:13.551060 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-c446k"] Nov 03 21:36:13.554581 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:36:13.554555 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c86a2a_007c_4a88_87cc_5a97de621e31.slice/crio-f932fa7ba50082b0bea70591bce4367389280a6a5acd874336d75aaae2c41a8a WatchSource:0}: Error finding container f932fa7ba50082b0bea70591bce4367389280a6a5acd874336d75aaae2c41a8a: Status 404 returned error can't find the container with id f932fa7ba50082b0bea70591bce4367389280a6a5acd874336d75aaae2c41a8a Nov 03 21:36:13.688328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:13.688287 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c446k" event={"ID":"b3c86a2a-007c-4a88-87cc-5a97de621e31","Type":"ContainerStarted","Data":"f932fa7ba50082b0bea70591bce4367389280a6a5acd874336d75aaae2c41a8a"} Nov 03 21:36:15.701046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:15.700998 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c446k" event={"ID":"b3c86a2a-007c-4a88-87cc-5a97de621e31","Type":"ContainerStarted","Data":"da5dc143a07181435f8f0fd51df41bc0abe3bb7695a76153e6df06aa34dc958f"} Nov 03 21:36:15.701460 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:15.701133 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c446k" Nov 03 21:36:15.744418 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:15.744111 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c446k" podStartSLOduration=2.08071547 podStartE2EDuration="3.74409442s" totalImagesPullingTime="1.66337895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:15.742612634 +0000 UTC m=+391.249840924" watchObservedRunningTime="2026-11-03 21:36:15.74409442 +0000 UTC m=+391.251322700" Nov 03 21:36:17.496502 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:17.496462 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6t5ln"] Nov 03 21:36:17.529848 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:17.529814 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6t5ln"] Nov 03 21:36:17.530012 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:17.529928 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6t5ln" Nov 03 21:36:17.549068 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:17.549042 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 21:36:17.549189 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:17.549103 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-sdqlc\"" Nov 03 21:36:17.710672 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:17.710631 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd4qs\" (UniqueName: \"kubernetes.io/projected/f03cef83-f82a-4b4c-901c-e091f6fc58dd-kube-api-access-nd4qs\") pod \"dns-operator-controller-manager-8557cbdb84-6t5ln\" (UID: \"f03cef83-f82a-4b4c-901c-e091f6fc58dd\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6t5ln" Nov 03 21:36:17.811709 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:17.811644 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd4qs\" (UniqueName: \"kubernetes.io/projected/f03cef83-f82a-4b4c-901c-e091f6fc58dd-kube-api-access-nd4qs\") pod \"dns-operator-controller-manager-8557cbdb84-6t5ln\" (UID: \"f03cef83-f82a-4b4c-901c-e091f6fc58dd\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6t5ln" Nov 03 21:36:17.834076 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:17.834055 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd4qs\" (UniqueName: \"kubernetes.io/projected/f03cef83-f82a-4b4c-901c-e091f6fc58dd-kube-api-access-nd4qs\") pod \"dns-operator-controller-manager-8557cbdb84-6t5ln\" (UID: \"f03cef83-f82a-4b4c-901c-e091f6fc58dd\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6t5ln" Nov 03 21:36:17.854430 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:17.854410 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6t5ln" Nov 03 21:36:18.143737 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:18.143656 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6t5ln"] Nov 03 21:36:18.149504 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:36:18.149475 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03cef83_f82a_4b4c_901c_e091f6fc58dd.slice/crio-80fefe0b8805cf2161d36a0f58092991b60ff3a76de9ad5c3b686ce0fa3ec620 WatchSource:0}: Error finding container 80fefe0b8805cf2161d36a0f58092991b60ff3a76de9ad5c3b686ce0fa3ec620: Status 404 returned error can't find the container with id 80fefe0b8805cf2161d36a0f58092991b60ff3a76de9ad5c3b686ce0fa3ec620 Nov 03 21:36:18.717871 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:18.717837 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6t5ln" event={"ID":"f03cef83-f82a-4b4c-901c-e091f6fc58dd","Type":"ContainerStarted","Data":"80fefe0b8805cf2161d36a0f58092991b60ff3a76de9ad5c3b686ce0fa3ec620"} Nov 03 21:36:20.730442 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:20.730409 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6t5ln" event={"ID":"f03cef83-f82a-4b4c-901c-e091f6fc58dd","Type":"ContainerStarted","Data":"39a3a1e1c1029b2bdec8a7f94c25365383116c06f7bec0bbdcd02c8228ab8bc0"} Nov 03 21:36:20.730848 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:20.730540 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6t5ln" Nov 03 21:36:20.773398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:20.773353 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6t5ln" podStartSLOduration=1.563620751 podStartE2EDuration="3.773340784s" totalImagesPullingTime="2.209720033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:20.771794693 +0000 UTC m=+396.279022973" watchObservedRunningTime="2026-11-03 21:36:20.773340784 +0000 UTC m=+396.280569061" Nov 03 21:36:21.680507 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:21.680472 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ghgpz" Nov 03 21:36:21.680704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:21.680572 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5754f75c9f-md9nn" Nov 03 21:36:21.680754 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:21.680712 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" Nov 03 21:36:22.645483 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:22.645448 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49"] Nov 03 21:36:22.648088 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:22.648070 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" Nov 03 21:36:22.677277 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:22.677253 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49"] Nov 03 21:36:22.752640 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:22.752603 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d41cc544-3a25-4973-aed1-dfe03f8f992b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qnw49\" (UID: \"d41cc544-3a25-4973-aed1-dfe03f8f992b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" Nov 03 21:36:22.752816 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:22.752713 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz5zr\" (UniqueName: \"kubernetes.io/projected/d41cc544-3a25-4973-aed1-dfe03f8f992b-kube-api-access-hz5zr\") pod \"kuadrant-operator-controller-manager-678d87766f-qnw49\" (UID: \"d41cc544-3a25-4973-aed1-dfe03f8f992b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" Nov 03 21:36:22.853809 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:22.853764 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz5zr\" (UniqueName: \"kubernetes.io/projected/d41cc544-3a25-4973-aed1-dfe03f8f992b-kube-api-access-hz5zr\") pod \"kuadrant-operator-controller-manager-678d87766f-qnw49\" (UID: \"d41cc544-3a25-4973-aed1-dfe03f8f992b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" Nov 03 21:36:22.853973 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:22.853877 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d41cc544-3a25-4973-aed1-dfe03f8f992b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qnw49\" (UID: \"d41cc544-3a25-4973-aed1-dfe03f8f992b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" Nov 03 21:36:22.854267 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:22.854247 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d41cc544-3a25-4973-aed1-dfe03f8f992b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qnw49\" (UID: \"d41cc544-3a25-4973-aed1-dfe03f8f992b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" Nov 03 21:36:22.878827 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:22.878797 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz5zr\" (UniqueName: \"kubernetes.io/projected/d41cc544-3a25-4973-aed1-dfe03f8f992b-kube-api-access-hz5zr\") pod \"kuadrant-operator-controller-manager-678d87766f-qnw49\" (UID: \"d41cc544-3a25-4973-aed1-dfe03f8f992b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" Nov 03 21:36:22.959672 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:22.959645 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" Nov 03 21:36:23.280197 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:23.280120 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49"] Nov 03 21:36:23.295088 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:36:23.295060 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41cc544_3a25_4973_aed1_dfe03f8f992b.slice/crio-67a4b72f8042f7a0d21f69c9f9acb808364b474dd31f03962eb88837213de335 WatchSource:0}: Error finding container 67a4b72f8042f7a0d21f69c9f9acb808364b474dd31f03962eb88837213de335: Status 404 returned error can't find the container with id 67a4b72f8042f7a0d21f69c9f9acb808364b474dd31f03962eb88837213de335 Nov 03 21:36:23.747872 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:23.747837 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" event={"ID":"d41cc544-3a25-4973-aed1-dfe03f8f992b","Type":"ContainerStarted","Data":"ea469bb20f1be010b4a5ebea794984372fb0bd5fb93acfb5e83cc8fc4c90a8ae"} Nov 03 21:36:23.748253 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:23.747892 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" event={"ID":"d41cc544-3a25-4973-aed1-dfe03f8f992b","Type":"ContainerStarted","Data":"67a4b72f8042f7a0d21f69c9f9acb808364b474dd31f03962eb88837213de335"} Nov 03 21:36:23.748253 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:23.747933 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" Nov 03 21:36:23.793174 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:23.793128 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" podStartSLOduration=1.793115711 podStartE2EDuration="1.793115711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:23.792170962 +0000 UTC m=+399.299399254" watchObservedRunningTime="2026-11-03 21:36:23.793115711 +0000 UTC m=+399.300343990" Nov 03 21:36:26.709204 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:26.709140 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c446k" Nov 03 21:36:31.737988 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:31.737959 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6t5ln" Nov 03 21:36:32.005982 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.005908 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs"] Nov 03 21:36:32.009614 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.009584 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.018250 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.018224 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982671c7-73d5-48b1-9067-d644e1b956d4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.018373 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.018274 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982671c7-73d5-48b1-9067-d644e1b956d4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.018438 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.018375 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982671c7-73d5-48b1-9067-d644e1b956d4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.018438 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.018423 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982671c7-73d5-48b1-9067-d644e1b956d4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.018530 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.018448 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982671c7-73d5-48b1-9067-d644e1b956d4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.018584 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.018551 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdtrl\" (UniqueName: \"kubernetes.io/projected/982671c7-73d5-48b1-9067-d644e1b956d4-kube-api-access-mdtrl\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.018641 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.018588 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982671c7-73d5-48b1-9067-d644e1b956d4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.018641 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.018623 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982671c7-73d5-48b1-9067-d644e1b956d4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.018786 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.018749 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982671c7-73d5-48b1-9067-d644e1b956d4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.043610 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.043574 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs"] Nov 03 21:36:32.051132 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.051105 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qx24q\"" Nov 03 21:36:32.119323 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.119287 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982671c7-73d5-48b1-9067-d644e1b956d4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.119588 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.119338 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982671c7-73d5-48b1-9067-d644e1b956d4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.119588 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.119361 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982671c7-73d5-48b1-9067-d644e1b956d4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.119588 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.119391 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdtrl\" (UniqueName: \"kubernetes.io/projected/982671c7-73d5-48b1-9067-d644e1b956d4-kube-api-access-mdtrl\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.119588 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.119416 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982671c7-73d5-48b1-9067-d644e1b956d4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.120011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.119620 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982671c7-73d5-48b1-9067-d644e1b956d4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.120011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.119850 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982671c7-73d5-48b1-9067-d644e1b956d4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.120127 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.120008 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982671c7-73d5-48b1-9067-d644e1b956d4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.120127 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.120025 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982671c7-73d5-48b1-9067-d644e1b956d4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.120127 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.120053 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982671c7-73d5-48b1-9067-d644e1b956d4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.120127 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.120093 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982671c7-73d5-48b1-9067-d644e1b956d4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.120127 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.120097 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982671c7-73d5-48b1-9067-d644e1b956d4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.120451 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.120397 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982671c7-73d5-48b1-9067-d644e1b956d4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.120714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.120689 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982671c7-73d5-48b1-9067-d644e1b956d4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.121706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.121654 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982671c7-73d5-48b1-9067-d644e1b956d4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.122425 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.122407 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982671c7-73d5-48b1-9067-d644e1b956d4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.159214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.159179 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982671c7-73d5-48b1-9067-d644e1b956d4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.159343 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.159303 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdtrl\" (UniqueName: \"kubernetes.io/projected/982671c7-73d5-48b1-9067-d644e1b956d4-kube-api-access-mdtrl\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs\" (UID: \"982671c7-73d5-48b1-9067-d644e1b956d4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.332155 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.332062 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:32.648551 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.648517 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs"] Nov 03 21:36:32.650648 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:36:32.650618 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982671c7_73d5_48b1_9067_d644e1b956d4.slice/crio-31448af4bb86b30b703fcb50cdde2c6204df35c7d5414685311ded8df0cb246f WatchSource:0}: Error finding container 31448af4bb86b30b703fcb50cdde2c6204df35c7d5414685311ded8df0cb246f: Status 404 returned error can't find the container with id 31448af4bb86b30b703fcb50cdde2c6204df35c7d5414685311ded8df0cb246f Nov 03 21:36:32.652889 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.652857 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:36:32.652947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.652924 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:36:32.652987 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.652951 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:36:32.796126 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.796072 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" event={"ID":"982671c7-73d5-48b1-9067-d644e1b956d4","Type":"ContainerStarted","Data":"f2e70142f4851f53fc228959d7121b3af68f90c30ec7d210b3ed2f319a776574"} Nov 03 21:36:32.796600 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.796156 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" event={"ID":"982671c7-73d5-48b1-9067-d644e1b956d4","Type":"ContainerStarted","Data":"31448af4bb86b30b703fcb50cdde2c6204df35c7d5414685311ded8df0cb246f"} Nov 03 21:36:32.842844 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:32.842785 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" podStartSLOduration=1.842769415 podStartE2EDuration="1.842769415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:32.84123599 +0000 UTC m=+408.348464270" watchObservedRunningTime="2026-11-03 21:36:32.842769415 +0000 UTC m=+408.349997695" Nov 03 21:36:33.332498 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:33.332468 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:33.337455 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:33.337425 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:33.801683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:33.801637 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:33.802453 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:33.802433 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs" Nov 03 21:36:34.755218 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:34.755191 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" Nov 03 21:36:34.845282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:34.845248 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v"] Nov 03 21:36:34.845799 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:34.845464 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" podUID="e2ea4359-b4cf-4827-9b65-86fa2cc39e60" containerName="manager" containerID="cri-o://db2db5705800cb4fa4da36b187eedc4de9ba9f854c794e8fd11e761bc56e0418" gracePeriod=10 Nov 03 21:36:36.005176 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:36.005145 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" Nov 03 21:36:36.050309 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:36.050274 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2ea4359-b4cf-4827-9b65-86fa2cc39e60-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2ea4359-b4cf-4827-9b65-86fa2cc39e60-extensions-socket-volume\") pod \"e2ea4359-b4cf-4827-9b65-86fa2cc39e60\" (UID: \"e2ea4359-b4cf-4827-9b65-86fa2cc39e60\") " Nov 03 21:36:36.050473 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:36.050334 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2ea4359-b4cf-4827-9b65-86fa2cc39e60-kube-api-access-rrcfm\" (UniqueName: \"kubernetes.io/projected/e2ea4359-b4cf-4827-9b65-86fa2cc39e60-kube-api-access-rrcfm\") pod \"e2ea4359-b4cf-4827-9b65-86fa2cc39e60\" (UID: \"e2ea4359-b4cf-4827-9b65-86fa2cc39e60\") " Nov 03 21:36:36.050848 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:36.050824 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ea4359-b4cf-4827-9b65-86fa2cc39e60-extensions-socket-volume" pod "e2ea4359-b4cf-4827-9b65-86fa2cc39e60" (UID: "e2ea4359-b4cf-4827-9b65-86fa2cc39e60"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:36:36.052574 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:36.052551 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ea4359-b4cf-4827-9b65-86fa2cc39e60-kube-api-access-rrcfm" pod "e2ea4359-b4cf-4827-9b65-86fa2cc39e60" (UID: "e2ea4359-b4cf-4827-9b65-86fa2cc39e60"). InnerVolumeSpecName "kube-api-access-rrcfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:36:36.151347 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:36.151258 5041 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2ea4359-b4cf-4827-9b65-86fa2cc39e60-extensions-socket-volume\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:36:36.151347 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:36.151292 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrcfm\" (UniqueName: \"kubernetes.io/projected/e2ea4359-b4cf-4827-9b65-86fa2cc39e60-kube-api-access-rrcfm\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:36:36.818634 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:36.818597 5041 generic.go:356] "Generic (PLEG): container finished" podID="e2ea4359-b4cf-4827-9b65-86fa2cc39e60" containerID="db2db5705800cb4fa4da36b187eedc4de9ba9f854c794e8fd11e761bc56e0418" exitCode=0 Nov 03 21:36:36.818837 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:36.818646 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" Nov 03 21:36:36.818837 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:36.818688 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" event={"ID":"e2ea4359-b4cf-4827-9b65-86fa2cc39e60","Type":"ContainerDied","Data":"db2db5705800cb4fa4da36b187eedc4de9ba9f854c794e8fd11e761bc56e0418"} Nov 03 21:36:36.818837 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:36.818729 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v" event={"ID":"e2ea4359-b4cf-4827-9b65-86fa2cc39e60","Type":"ContainerDied","Data":"eb49609f8f7dd97fd686b21a3f4deea0b4ffa70d58a2002cda4a9df60e6eef03"} Nov 03 21:36:36.818837 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:36.818751 5041 scope.go:122] "RemoveContainer" containerID="db2db5705800cb4fa4da36b187eedc4de9ba9f854c794e8fd11e761bc56e0418" Nov 03 21:36:36.827709 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:36.827679 5041 scope.go:122] "RemoveContainer" containerID="db2db5705800cb4fa4da36b187eedc4de9ba9f854c794e8fd11e761bc56e0418" Nov 03 21:36:36.828004 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:36:36.827984 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db2db5705800cb4fa4da36b187eedc4de9ba9f854c794e8fd11e761bc56e0418\": container with ID starting with db2db5705800cb4fa4da36b187eedc4de9ba9f854c794e8fd11e761bc56e0418 not found: ID does not exist" containerID="db2db5705800cb4fa4da36b187eedc4de9ba9f854c794e8fd11e761bc56e0418" Nov 03 21:36:36.828054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:36.828012 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db2db5705800cb4fa4da36b187eedc4de9ba9f854c794e8fd11e761bc56e0418"} err="failed to get container status \"db2db5705800cb4fa4da36b187eedc4de9ba9f854c794e8fd11e761bc56e0418\": rpc error: code = NotFound desc = could not find container \"db2db5705800cb4fa4da36b187eedc4de9ba9f854c794e8fd11e761bc56e0418\": container with ID starting with db2db5705800cb4fa4da36b187eedc4de9ba9f854c794e8fd11e761bc56e0418 not found: ID does not exist" Nov 03 21:36:36.887831 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:36.887747 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v"] Nov 03 21:36:36.901372 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:36.901343 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-ptj5v"] Nov 03 21:36:38.620679 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:38.620625 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2ea4359-b4cf-4827-9b65-86fa2cc39e60" path="/var/lib/kubelet/pods/e2ea4359-b4cf-4827-9b65-86fa2cc39e60/volumes" Nov 03 21:36:39.186469 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.186433 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr"] Nov 03 21:36:39.186997 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.186895 5041 cpu_manager.go:414] "containerMap: removing container" podUID="e2ea4359-b4cf-4827-9b65-86fa2cc39e60" containerName="manager" Nov 03 21:36:39.186997 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.186920 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2ea4359-b4cf-4827-9b65-86fa2cc39e60" containerName="manager" Nov 03 21:36:39.187156 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.187044 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2ea4359-b4cf-4827-9b65-86fa2cc39e60" containerName="manager" Nov 03 21:36:39.189258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.189239 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" Nov 03 21:36:39.205287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.205263 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 21:36:39.219098 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.219061 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-vjwwh\"" Nov 03 21:36:39.219227 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.219105 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 21:36:39.247553 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.247519 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 21:36:39.249629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.249609 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr"] Nov 03 21:36:39.274091 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.274060 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbq7v\" (UniqueName: \"kubernetes.io/projected/e83c023b-6164-4f13-89bd-9553952667b7-kube-api-access-tbq7v\") pod \"kserve-module-controller-manager-7566cf5fc8-kvrbr\" (UID: \"e83c023b-6164-4f13-89bd-9553952667b7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" Nov 03 21:36:39.274216 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.274120 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e83c023b-6164-4f13-89bd-9553952667b7-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kvrbr\" (UID: \"e83c023b-6164-4f13-89bd-9553952667b7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" Nov 03 21:36:39.274216 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.274204 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e83c023b-6164-4f13-89bd-9553952667b7-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kvrbr\" (UID: \"e83c023b-6164-4f13-89bd-9553952667b7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" Nov 03 21:36:39.274283 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.274244 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e83c023b-6164-4f13-89bd-9553952667b7-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kvrbr\" (UID: \"e83c023b-6164-4f13-89bd-9553952667b7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" Nov 03 21:36:39.374900 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.374860 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e83c023b-6164-4f13-89bd-9553952667b7-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kvrbr\" (UID: \"e83c023b-6164-4f13-89bd-9553952667b7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" Nov 03 21:36:39.375103 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.374919 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbq7v\" (UniqueName: \"kubernetes.io/projected/e83c023b-6164-4f13-89bd-9553952667b7-kube-api-access-tbq7v\") pod \"kserve-module-controller-manager-7566cf5fc8-kvrbr\" (UID: \"e83c023b-6164-4f13-89bd-9553952667b7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" Nov 03 21:36:39.375103 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.374964 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e83c023b-6164-4f13-89bd-9553952667b7-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kvrbr\" (UID: \"e83c023b-6164-4f13-89bd-9553952667b7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" Nov 03 21:36:39.375103 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.375000 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e83c023b-6164-4f13-89bd-9553952667b7-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kvrbr\" (UID: \"e83c023b-6164-4f13-89bd-9553952667b7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" Nov 03 21:36:39.375371 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.375347 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e83c023b-6164-4f13-89bd-9553952667b7-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-kvrbr\" (UID: \"e83c023b-6164-4f13-89bd-9553952667b7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" Nov 03 21:36:39.375441 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.375389 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e83c023b-6164-4f13-89bd-9553952667b7-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-kvrbr\" (UID: \"e83c023b-6164-4f13-89bd-9553952667b7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" Nov 03 21:36:39.375832 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.375815 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e83c023b-6164-4f13-89bd-9553952667b7-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-kvrbr\" (UID: \"e83c023b-6164-4f13-89bd-9553952667b7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" Nov 03 21:36:39.395566 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.395541 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbq7v\" (UniqueName: \"kubernetes.io/projected/e83c023b-6164-4f13-89bd-9553952667b7-kube-api-access-tbq7v\") pod \"kserve-module-controller-manager-7566cf5fc8-kvrbr\" (UID: \"e83c023b-6164-4f13-89bd-9553952667b7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" Nov 03 21:36:39.501816 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.501780 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" Nov 03 21:36:39.805483 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.805459 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr"] Nov 03 21:36:39.807651 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:36:39.807613 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83c023b_6164_4f13_89bd_9553952667b7.slice/crio-7a7a749e772567159c0ab64e223093529075cb8bd499831b51554fe8a514f3ef WatchSource:0}: Error finding container 7a7a749e772567159c0ab64e223093529075cb8bd499831b51554fe8a514f3ef: Status 404 returned error can't find the container with id 7a7a749e772567159c0ab64e223093529075cb8bd499831b51554fe8a514f3ef Nov 03 21:36:39.836742 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:39.836692 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" event={"ID":"e83c023b-6164-4f13-89bd-9553952667b7","Type":"ContainerStarted","Data":"7a7a749e772567159c0ab64e223093529075cb8bd499831b51554fe8a514f3ef"} Nov 03 21:36:40.786097 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:36:40.786050 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:36:42.857336 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:42.857303 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" event={"ID":"e83c023b-6164-4f13-89bd-9553952667b7","Type":"ContainerStarted","Data":"028aa1e837220cfa70e2ab5a190ca036472fd02aabf6d8efa0900ab312adfeea"} Nov 03 21:36:42.857714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:42.857364 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" Nov 03 21:36:42.899704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:42.899639 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" podStartSLOduration=0.98997093 podStartE2EDuration="3.899626547s" totalImagesPullingTime="2.909655617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:42.898450327 +0000 UTC m=+418.405678606" watchObservedRunningTime="2026-11-03 21:36:42.899626547 +0000 UTC m=+418.406854824" Nov 03 21:36:54.548857 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:54.548818 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-hzchq"] Nov 03 21:36:54.554750 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:54.554726 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hzchq" Nov 03 21:36:54.569545 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:54.569513 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-hzchq"] Nov 03 21:36:54.581340 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:54.581304 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:36:54.581340 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:54.581316 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:36:54.584454 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:54.584419 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 21:36:54.584780 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:54.584476 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fqshd\"" Nov 03 21:36:54.596468 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:54.596435 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gds4\" (UniqueName: \"kubernetes.io/projected/c7500df4-c2fc-47e5-bcbe-330cf017db34-kube-api-access-8gds4\") pod \"postgres-7cdbf6dc44-hzchq\" (UID: \"c7500df4-c2fc-47e5-bcbe-330cf017db34\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hzchq" Nov 03 21:36:54.596694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:54.596579 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c7500df4-c2fc-47e5-bcbe-330cf017db34-data\") pod \"postgres-7cdbf6dc44-hzchq\" (UID: \"c7500df4-c2fc-47e5-bcbe-330cf017db34\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hzchq" Nov 03 21:36:54.697131 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:54.697091 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gds4\" (UniqueName: \"kubernetes.io/projected/c7500df4-c2fc-47e5-bcbe-330cf017db34-kube-api-access-8gds4\") pod \"postgres-7cdbf6dc44-hzchq\" (UID: \"c7500df4-c2fc-47e5-bcbe-330cf017db34\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hzchq" Nov 03 21:36:54.697310 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:54.697195 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c7500df4-c2fc-47e5-bcbe-330cf017db34-data\") pod \"postgres-7cdbf6dc44-hzchq\" (UID: \"c7500df4-c2fc-47e5-bcbe-330cf017db34\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hzchq" Nov 03 21:36:54.697554 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:54.697537 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c7500df4-c2fc-47e5-bcbe-330cf017db34-data\") pod \"postgres-7cdbf6dc44-hzchq\" (UID: \"c7500df4-c2fc-47e5-bcbe-330cf017db34\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hzchq" Nov 03 21:36:54.718245 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:54.718211 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gds4\" (UniqueName: \"kubernetes.io/projected/c7500df4-c2fc-47e5-bcbe-330cf017db34-kube-api-access-8gds4\") pod \"postgres-7cdbf6dc44-hzchq\" (UID: \"c7500df4-c2fc-47e5-bcbe-330cf017db34\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hzchq" Nov 03 21:36:54.867826 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:54.867727 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hzchq" Nov 03 21:36:55.163195 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:55.163166 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-hzchq"] Nov 03 21:36:55.165674 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:36:55.165630 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7500df4_c2fc_47e5_bcbe_330cf017db34.slice/crio-afce2bc53e7c6dab2daf4cd8a3b083f91fc8f9441ed8861ed6d3f5d71fefb6fe WatchSource:0}: Error finding container afce2bc53e7c6dab2daf4cd8a3b083f91fc8f9441ed8861ed6d3f5d71fefb6fe: Status 404 returned error can't find the container with id afce2bc53e7c6dab2daf4cd8a3b083f91fc8f9441ed8861ed6d3f5d71fefb6fe Nov 03 21:36:55.922922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:36:55.922883 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hzchq" event={"ID":"c7500df4-c2fc-47e5-bcbe-330cf017db34","Type":"ContainerStarted","Data":"afce2bc53e7c6dab2daf4cd8a3b083f91fc8f9441ed8861ed6d3f5d71fefb6fe"} Nov 03 21:37:00.951308 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:00.951266 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hzchq" event={"ID":"c7500df4-c2fc-47e5-bcbe-330cf017db34","Type":"ContainerStarted","Data":"2c1055aedbafc3fbe8460d2d40c3230a9b905e44596e12036976d4775ed3c593"} Nov 03 21:37:00.951788 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:00.951357 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hzchq" Nov 03 21:37:00.991472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:00.991408 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hzchq" podStartSLOduration=2.044335978 podStartE2EDuration="6.991391381s" totalImagesPullingTime="4.947055403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:00.990378184 +0000 UTC m=+436.497606462" watchObservedRunningTime="2026-11-03 21:37:00.991391381 +0000 UTC m=+436.498619661" Nov 03 21:37:03.864178 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:03.864150 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-kvrbr" Nov 03 21:37:06.985888 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:06.985859 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hzchq" Nov 03 21:37:10.775900 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:37:10.775857 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:37:13.892212 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:13.892168 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-smhz7"] Nov 03 21:37:13.896993 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:13.896964 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-smhz7" Nov 03 21:37:13.915189 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:13.915154 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-smhz7"] Nov 03 21:37:13.927924 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:13.927889 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 21:37:13.928101 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:13.928032 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 21:37:13.928142 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:13.928103 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 21:37:13.928251 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:13.928217 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cknnf\"" Nov 03 21:37:13.970339 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:13.970286 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zqxr\" (UniqueName: \"kubernetes.io/projected/4dee3593-c5f5-4555-9541-332caf3f00cd-kube-api-access-2zqxr\") pod \"maas-controller-579cf7c6d-smhz7\" (UID: \"4dee3593-c5f5-4555-9541-332caf3f00cd\") " pod="opendatahub/maas-controller-579cf7c6d-smhz7" Nov 03 21:37:13.970532 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:13.970354 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dee3593-c5f5-4555-9541-332caf3f00cd-webhook-cert\") pod \"maas-controller-579cf7c6d-smhz7\" (UID: \"4dee3593-c5f5-4555-9541-332caf3f00cd\") " pod="opendatahub/maas-controller-579cf7c6d-smhz7" Nov 03 21:37:13.970532 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:13.970506 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dee3593-c5f5-4555-9541-332caf3f00cd-metrics-certs\") pod \"maas-controller-579cf7c6d-smhz7\" (UID: \"4dee3593-c5f5-4555-9541-332caf3f00cd\") " pod="opendatahub/maas-controller-579cf7c6d-smhz7" Nov 03 21:37:14.071389 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.071339 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zqxr\" (UniqueName: \"kubernetes.io/projected/4dee3593-c5f5-4555-9541-332caf3f00cd-kube-api-access-2zqxr\") pod \"maas-controller-579cf7c6d-smhz7\" (UID: \"4dee3593-c5f5-4555-9541-332caf3f00cd\") " pod="opendatahub/maas-controller-579cf7c6d-smhz7" Nov 03 21:37:14.071389 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.071397 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dee3593-c5f5-4555-9541-332caf3f00cd-webhook-cert\") pod \"maas-controller-579cf7c6d-smhz7\" (UID: \"4dee3593-c5f5-4555-9541-332caf3f00cd\") " pod="opendatahub/maas-controller-579cf7c6d-smhz7" Nov 03 21:37:14.071610 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.071444 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dee3593-c5f5-4555-9541-332caf3f00cd-metrics-certs\") pod \"maas-controller-579cf7c6d-smhz7\" (UID: \"4dee3593-c5f5-4555-9541-332caf3f00cd\") " pod="opendatahub/maas-controller-579cf7c6d-smhz7" Nov 03 21:37:14.073924 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.073895 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dee3593-c5f5-4555-9541-332caf3f00cd-webhook-cert\") pod \"maas-controller-579cf7c6d-smhz7\" (UID: \"4dee3593-c5f5-4555-9541-332caf3f00cd\") " pod="opendatahub/maas-controller-579cf7c6d-smhz7" Nov 03 21:37:14.074059 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.074008 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dee3593-c5f5-4555-9541-332caf3f00cd-metrics-certs\") pod \"maas-controller-579cf7c6d-smhz7\" (UID: \"4dee3593-c5f5-4555-9541-332caf3f00cd\") " pod="opendatahub/maas-controller-579cf7c6d-smhz7" Nov 03 21:37:14.091887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.091853 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zqxr\" (UniqueName: \"kubernetes.io/projected/4dee3593-c5f5-4555-9541-332caf3f00cd-kube-api-access-2zqxr\") pod \"maas-controller-579cf7c6d-smhz7\" (UID: \"4dee3593-c5f5-4555-9541-332caf3f00cd\") " pod="opendatahub/maas-controller-579cf7c6d-smhz7" Nov 03 21:37:14.210674 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.210627 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-smhz7" Nov 03 21:37:14.215884 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.215858 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d575fdb56-8br4h"] Nov 03 21:37:14.219699 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.219675 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:37:14.239848 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.239810 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d575fdb56-8br4h"] Nov 03 21:37:14.275034 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.274262 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82b63ac-65d4-4e30-8b51-d308cb3d335d-metrics-certs\") pod \"maas-controller-6d575fdb56-8br4h\" (UID: \"d82b63ac-65d4-4e30-8b51-d308cb3d335d\") " pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:37:14.275034 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.274365 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2882x\" (UniqueName: \"kubernetes.io/projected/d82b63ac-65d4-4e30-8b51-d308cb3d335d-kube-api-access-2882x\") pod \"maas-controller-6d575fdb56-8br4h\" (UID: \"d82b63ac-65d4-4e30-8b51-d308cb3d335d\") " pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:37:14.275034 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.274398 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d82b63ac-65d4-4e30-8b51-d308cb3d335d-webhook-cert\") pod \"maas-controller-6d575fdb56-8br4h\" (UID: \"d82b63ac-65d4-4e30-8b51-d308cb3d335d\") " pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:37:14.375654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.375603 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82b63ac-65d4-4e30-8b51-d308cb3d335d-metrics-certs\") pod \"maas-controller-6d575fdb56-8br4h\" (UID: \"d82b63ac-65d4-4e30-8b51-d308cb3d335d\") " pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:37:14.375870 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.375704 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2882x\" (UniqueName: \"kubernetes.io/projected/d82b63ac-65d4-4e30-8b51-d308cb3d335d-kube-api-access-2882x\") pod \"maas-controller-6d575fdb56-8br4h\" (UID: \"d82b63ac-65d4-4e30-8b51-d308cb3d335d\") " pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:37:14.375870 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.375723 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d82b63ac-65d4-4e30-8b51-d308cb3d335d-webhook-cert\") pod \"maas-controller-6d575fdb56-8br4h\" (UID: \"d82b63ac-65d4-4e30-8b51-d308cb3d335d\") " pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:37:14.378134 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.378108 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82b63ac-65d4-4e30-8b51-d308cb3d335d-metrics-certs\") pod \"maas-controller-6d575fdb56-8br4h\" (UID: \"d82b63ac-65d4-4e30-8b51-d308cb3d335d\") " pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:37:14.378134 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.378133 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d82b63ac-65d4-4e30-8b51-d308cb3d335d-webhook-cert\") pod \"maas-controller-6d575fdb56-8br4h\" (UID: \"d82b63ac-65d4-4e30-8b51-d308cb3d335d\") " pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:37:14.396212 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.396180 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2882x\" (UniqueName: \"kubernetes.io/projected/d82b63ac-65d4-4e30-8b51-d308cb3d335d-kube-api-access-2882x\") pod \"maas-controller-6d575fdb56-8br4h\" (UID: \"d82b63ac-65d4-4e30-8b51-d308cb3d335d\") " pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:37:14.519481 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.519444 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-smhz7"] Nov 03 21:37:14.521172 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:37:14.521142 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dee3593_c5f5_4555_9541_332caf3f00cd.slice/crio-f8bd1b4ae523ea84e86cfc428c09340e1d000d0f0db996d50da4a85ba51e5fdd WatchSource:0}: Error finding container f8bd1b4ae523ea84e86cfc428c09340e1d000d0f0db996d50da4a85ba51e5fdd: Status 404 returned error can't find the container with id f8bd1b4ae523ea84e86cfc428c09340e1d000d0f0db996d50da4a85ba51e5fdd Nov 03 21:37:14.553771 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.553736 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:37:14.849951 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:14.849924 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d575fdb56-8br4h"] Nov 03 21:37:14.851841 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:37:14.851806 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82b63ac_65d4_4e30_8b51_d308cb3d335d.slice/crio-29904215fd7aa852143854a01d84967dd972a4cda5ad63b57373068817fdc8b3 WatchSource:0}: Error finding container 29904215fd7aa852143854a01d84967dd972a4cda5ad63b57373068817fdc8b3: Status 404 returned error can't find the container with id 29904215fd7aa852143854a01d84967dd972a4cda5ad63b57373068817fdc8b3 Nov 03 21:37:15.022759 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:15.022712 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-smhz7" event={"ID":"4dee3593-c5f5-4555-9541-332caf3f00cd","Type":"ContainerStarted","Data":"f8bd1b4ae523ea84e86cfc428c09340e1d000d0f0db996d50da4a85ba51e5fdd"} Nov 03 21:37:15.023964 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:15.023939 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d575fdb56-8br4h" event={"ID":"d82b63ac-65d4-4e30-8b51-d308cb3d335d","Type":"ContainerStarted","Data":"29904215fd7aa852143854a01d84967dd972a4cda5ad63b57373068817fdc8b3"} Nov 03 21:37:18.045484 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:18.045449 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-smhz7" event={"ID":"4dee3593-c5f5-4555-9541-332caf3f00cd","Type":"ContainerStarted","Data":"43e4bb87cc265d9ae656a0f57fe6a72a9fd1d0895cb3f21131963c05a2f3dd6e"} Nov 03 21:37:18.045969 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:18.045519 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-smhz7" Nov 03 21:37:18.047013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:18.046987 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d575fdb56-8br4h" event={"ID":"d82b63ac-65d4-4e30-8b51-d308cb3d335d","Type":"ContainerStarted","Data":"e6f611effcbffbd5ced79fc870a015b984cc5d508dc3259d208460ee36d03206"} Nov 03 21:37:18.047146 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:18.047050 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:37:18.101102 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:18.101052 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-smhz7" podStartSLOduration=2.231281325 podStartE2EDuration="5.101036826s" totalImagesPullingTime="2.869755501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:18.099530126 +0000 UTC m=+453.606758408" watchObservedRunningTime="2026-11-03 21:37:18.101036826 +0000 UTC m=+453.608265107" Nov 03 21:37:18.146878 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:18.146820 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6d575fdb56-8br4h" podStartSLOduration=1.454861347 podStartE2EDuration="4.146802259s" totalImagesPullingTime="2.691940912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:18.146549337 +0000 UTC m=+453.653777660" watchObservedRunningTime="2026-11-03 21:37:18.146802259 +0000 UTC m=+453.654030540" Nov 03 21:37:19.040079 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.040045 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7"] Nov 03 21:37:19.043992 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.043972 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:19.063217 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.063186 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7"] Nov 03 21:37:19.073285 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.073263 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 21:37:19.073416 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.073271 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 21:37:19.073483 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.073426 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-nvbmg\"" Nov 03 21:37:19.073574 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.073550 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 21:37:19.120677 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.120623 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1dc05e-ba05-4715-9236-1eabf21fb915-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-95xs7\" (UID: \"8c1dc05e-ba05-4715-9236-1eabf21fb915\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:19.120839 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.120802 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/8c1dc05e-ba05-4715-9236-1eabf21fb915-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-95xs7\" (UID: \"8c1dc05e-ba05-4715-9236-1eabf21fb915\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:19.120914 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.120858 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8c1dc05e-ba05-4715-9236-1eabf21fb915-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-95xs7\" (UID: \"8c1dc05e-ba05-4715-9236-1eabf21fb915\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:19.120914 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.120903 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kg2h\" (UniqueName: \"kubernetes.io/projected/8c1dc05e-ba05-4715-9236-1eabf21fb915-kube-api-access-7kg2h\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-95xs7\" (UID: \"8c1dc05e-ba05-4715-9236-1eabf21fb915\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:19.221723 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.221677 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1dc05e-ba05-4715-9236-1eabf21fb915-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-95xs7\" (UID: \"8c1dc05e-ba05-4715-9236-1eabf21fb915\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:19.221924 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.221768 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/8c1dc05e-ba05-4715-9236-1eabf21fb915-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-95xs7\" (UID: \"8c1dc05e-ba05-4715-9236-1eabf21fb915\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:19.221924 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.221796 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8c1dc05e-ba05-4715-9236-1eabf21fb915-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-95xs7\" (UID: \"8c1dc05e-ba05-4715-9236-1eabf21fb915\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:19.221924 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.221820 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kg2h\" (UniqueName: \"kubernetes.io/projected/8c1dc05e-ba05-4715-9236-1eabf21fb915-kube-api-access-7kg2h\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-95xs7\" (UID: \"8c1dc05e-ba05-4715-9236-1eabf21fb915\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:19.222102 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:37:19.221954 5041 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 03 21:37:19.222102 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:37:19.222022 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c1dc05e-ba05-4715-9236-1eabf21fb915-tls-cert podName:8c1dc05e-ba05-4715-9236-1eabf21fb915 nodeName:}" failed. No retries permitted until 2026-11-03 21:37:19.722002681 +0000 UTC m=+455.229230953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/8c1dc05e-ba05-4715-9236-1eabf21fb915-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-95xs7" (UID: "8c1dc05e-ba05-4715-9236-1eabf21fb915") : secret "usage-logs-tenancy-proxy-tls" not found Nov 03 21:37:19.222448 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.222420 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1dc05e-ba05-4715-9236-1eabf21fb915-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-95xs7\" (UID: \"8c1dc05e-ba05-4715-9236-1eabf21fb915\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:19.222722 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.222699 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/8c1dc05e-ba05-4715-9236-1eabf21fb915-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-95xs7\" (UID: \"8c1dc05e-ba05-4715-9236-1eabf21fb915\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:19.243275 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.243236 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kg2h\" (UniqueName: \"kubernetes.io/projected/8c1dc05e-ba05-4715-9236-1eabf21fb915-kube-api-access-7kg2h\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-95xs7\" (UID: \"8c1dc05e-ba05-4715-9236-1eabf21fb915\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:19.727016 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.726969 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8c1dc05e-ba05-4715-9236-1eabf21fb915-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-95xs7\" (UID: \"8c1dc05e-ba05-4715-9236-1eabf21fb915\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:19.736380 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.736351 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8c1dc05e-ba05-4715-9236-1eabf21fb915-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-95xs7\" (UID: \"8c1dc05e-ba05-4715-9236-1eabf21fb915\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:19.957824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:19.957795 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:20.254615 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:20.254587 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7"] Nov 03 21:37:20.256400 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:37:20.256368 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1dc05e_ba05_4715_9236_1eabf21fb915.slice/crio-508ff04f0ed11c10790a12243d88bd2b85eb145d4bb410079171c18533cc13f3 WatchSource:0}: Error finding container 508ff04f0ed11c10790a12243d88bd2b85eb145d4bb410079171c18533cc13f3: Status 404 returned error can't find the container with id 508ff04f0ed11c10790a12243d88bd2b85eb145d4bb410079171c18533cc13f3 Nov 03 21:37:21.064596 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:21.064564 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" event={"ID":"8c1dc05e-ba05-4715-9236-1eabf21fb915","Type":"ContainerStarted","Data":"508ff04f0ed11c10790a12243d88bd2b85eb145d4bb410079171c18533cc13f3"} Nov 03 21:37:29.056051 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:29.056006 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:37:29.056698 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:29.056204 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-smhz7" Nov 03 21:37:29.128045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:29.128011 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-smhz7"] Nov 03 21:37:29.128235 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:29.128212 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-smhz7" podUID="4dee3593-c5f5-4555-9541-332caf3f00cd" containerName="manager" containerID="cri-o://43e4bb87cc265d9ae656a0f57fe6a72a9fd1d0895cb3f21131963c05a2f3dd6e" gracePeriod=30 Nov 03 21:37:29.411317 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:29.411284 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-smhz7" Nov 03 21:37:29.524890 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:29.524851 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dee3593-c5f5-4555-9541-332caf3f00cd-webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dee3593-c5f5-4555-9541-332caf3f00cd-webhook-cert\") pod \"4dee3593-c5f5-4555-9541-332caf3f00cd\" (UID: \"4dee3593-c5f5-4555-9541-332caf3f00cd\") " Nov 03 21:37:29.525097 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:29.524938 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dee3593-c5f5-4555-9541-332caf3f00cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dee3593-c5f5-4555-9541-332caf3f00cd-metrics-certs\") pod \"4dee3593-c5f5-4555-9541-332caf3f00cd\" (UID: \"4dee3593-c5f5-4555-9541-332caf3f00cd\") " Nov 03 21:37:29.525097 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:29.524989 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dee3593-c5f5-4555-9541-332caf3f00cd-kube-api-access-2zqxr\" (UniqueName: \"kubernetes.io/projected/4dee3593-c5f5-4555-9541-332caf3f00cd-kube-api-access-2zqxr\") pod \"4dee3593-c5f5-4555-9541-332caf3f00cd\" (UID: \"4dee3593-c5f5-4555-9541-332caf3f00cd\") " Nov 03 21:37:29.528277 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:29.528232 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dee3593-c5f5-4555-9541-332caf3f00cd-kube-api-access-2zqxr" pod "4dee3593-c5f5-4555-9541-332caf3f00cd" (UID: "4dee3593-c5f5-4555-9541-332caf3f00cd"). InnerVolumeSpecName "kube-api-access-2zqxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:37:29.528448 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:29.528307 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dee3593-c5f5-4555-9541-332caf3f00cd-webhook-cert" pod "4dee3593-c5f5-4555-9541-332caf3f00cd" (UID: "4dee3593-c5f5-4555-9541-332caf3f00cd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:37:29.529196 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:29.529158 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dee3593-c5f5-4555-9541-332caf3f00cd-metrics-certs" pod "4dee3593-c5f5-4555-9541-332caf3f00cd" (UID: "4dee3593-c5f5-4555-9541-332caf3f00cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:37:29.626005 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:29.625917 5041 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dee3593-c5f5-4555-9541-332caf3f00cd-webhook-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:37:29.626005 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:29.625958 5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dee3593-c5f5-4555-9541-332caf3f00cd-metrics-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:37:29.626005 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:29.625968 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zqxr\" (UniqueName: \"kubernetes.io/projected/4dee3593-c5f5-4555-9541-332caf3f00cd-kube-api-access-2zqxr\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:37:30.128404 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:30.128365 5041 generic.go:356] "Generic (PLEG): container finished" podID="4dee3593-c5f5-4555-9541-332caf3f00cd" containerID="43e4bb87cc265d9ae656a0f57fe6a72a9fd1d0895cb3f21131963c05a2f3dd6e" exitCode=0 Nov 03 21:37:30.128878 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:30.128422 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-smhz7" Nov 03 21:37:30.128878 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:30.128454 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-smhz7" event={"ID":"4dee3593-c5f5-4555-9541-332caf3f00cd","Type":"ContainerDied","Data":"43e4bb87cc265d9ae656a0f57fe6a72a9fd1d0895cb3f21131963c05a2f3dd6e"} Nov 03 21:37:30.128878 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:30.128518 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-smhz7" event={"ID":"4dee3593-c5f5-4555-9541-332caf3f00cd","Type":"ContainerDied","Data":"f8bd1b4ae523ea84e86cfc428c09340e1d000d0f0db996d50da4a85ba51e5fdd"} Nov 03 21:37:30.128878 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:30.128543 5041 scope.go:122] "RemoveContainer" containerID="43e4bb87cc265d9ae656a0f57fe6a72a9fd1d0895cb3f21131963c05a2f3dd6e" Nov 03 21:37:30.139192 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:30.139027 5041 scope.go:122] "RemoveContainer" containerID="43e4bb87cc265d9ae656a0f57fe6a72a9fd1d0895cb3f21131963c05a2f3dd6e" Nov 03 21:37:30.139379 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:37:30.139355 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e4bb87cc265d9ae656a0f57fe6a72a9fd1d0895cb3f21131963c05a2f3dd6e\": container with ID starting with 43e4bb87cc265d9ae656a0f57fe6a72a9fd1d0895cb3f21131963c05a2f3dd6e not found: ID does not exist" containerID="43e4bb87cc265d9ae656a0f57fe6a72a9fd1d0895cb3f21131963c05a2f3dd6e" Nov 03 21:37:30.139487 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:30.139387 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e4bb87cc265d9ae656a0f57fe6a72a9fd1d0895cb3f21131963c05a2f3dd6e"} err="failed to get container status \"43e4bb87cc265d9ae656a0f57fe6a72a9fd1d0895cb3f21131963c05a2f3dd6e\": rpc error: code = NotFound desc = could not find container \"43e4bb87cc265d9ae656a0f57fe6a72a9fd1d0895cb3f21131963c05a2f3dd6e\": container with ID starting with 43e4bb87cc265d9ae656a0f57fe6a72a9fd1d0895cb3f21131963c05a2f3dd6e not found: ID does not exist" Nov 03 21:37:30.202544 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:30.202502 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-smhz7"] Nov 03 21:37:30.215955 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:30.215920 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-smhz7"] Nov 03 21:37:30.622979 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:30.622937 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dee3593-c5f5-4555-9541-332caf3f00cd" path="/var/lib/kubelet/pods/4dee3593-c5f5-4555-9541-332caf3f00cd/volumes" Nov 03 21:37:32.150006 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:32.149968 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" event={"ID":"8c1dc05e-ba05-4715-9236-1eabf21fb915","Type":"ContainerStarted","Data":"c5f39d0cfd57305e7e49420b1bb11af869c903743e633699b9d7fc163a916c71"} Nov 03 21:37:32.150384 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:32.150094 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:32.194175 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:32.194064 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" podStartSLOduration=2.113659055 podStartE2EDuration="13.194048323s" totalImagesPullingTime="11.080389268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:32.19215756 +0000 UTC m=+467.699385841" watchObservedRunningTime="2026-11-03 21:37:32.194048323 +0000 UTC m=+467.701276607" Nov 03 21:37:38.162928 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.162892 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lrr9z"] Nov 03 21:37:38.163509 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.163340 5041 cpu_manager.go:414] "containerMap: removing container" podUID="4dee3593-c5f5-4555-9541-332caf3f00cd" containerName="manager" Nov 03 21:37:38.163509 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.163353 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dee3593-c5f5-4555-9541-332caf3f00cd" containerName="manager" Nov 03 21:37:38.163509 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.163411 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dee3593-c5f5-4555-9541-332caf3f00cd" containerName="manager" Nov 03 21:37:38.194498 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.194460 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lrr9z"] Nov 03 21:37:38.194708 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.194613 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-lrr9z" Nov 03 21:37:38.201022 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.200989 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/416c8af4-2b21-4015-9cda-e4a7f75d8506-cert\") pod \"kserve-controller-manager-776cc66d47-lrr9z\" (UID: \"416c8af4-2b21-4015-9cda-e4a7f75d8506\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lrr9z" Nov 03 21:37:38.201216 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.201131 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpf7x\" (UniqueName: \"kubernetes.io/projected/416c8af4-2b21-4015-9cda-e4a7f75d8506-kube-api-access-xpf7x\") pod \"kserve-controller-manager-776cc66d47-lrr9z\" (UID: \"416c8af4-2b21-4015-9cda-e4a7f75d8506\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lrr9z" Nov 03 21:37:38.214794 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.214760 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 21:37:38.228194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.228157 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qrd6h\"" Nov 03 21:37:38.248959 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.248924 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vppdx"] Nov 03 21:37:38.251860 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.251837 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-vppdx" Nov 03 21:37:38.269807 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.269772 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vppdx"] Nov 03 21:37:38.291704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.291649 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-2cctc"] Nov 03 21:37:38.294896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.294872 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-2cctc" Nov 03 21:37:38.301749 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.301716 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/416c8af4-2b21-4015-9cda-e4a7f75d8506-cert\") pod \"kserve-controller-manager-776cc66d47-lrr9z\" (UID: \"416c8af4-2b21-4015-9cda-e4a7f75d8506\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lrr9z" Nov 03 21:37:38.301842 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.301774 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvbh7\" (UniqueName: \"kubernetes.io/projected/a8f799a1-8427-44a3-a495-74720e82292a-kube-api-access-rvbh7\") pod \"model-serving-api-75bd89fdfc-vppdx\" (UID: \"a8f799a1-8427-44a3-a495-74720e82292a\") " pod="opendatahub/model-serving-api-75bd89fdfc-vppdx" Nov 03 21:37:38.301910 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.301836 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpf7x\" (UniqueName: \"kubernetes.io/projected/416c8af4-2b21-4015-9cda-e4a7f75d8506-kube-api-access-xpf7x\") pod \"kserve-controller-manager-776cc66d47-lrr9z\" (UID: \"416c8af4-2b21-4015-9cda-e4a7f75d8506\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lrr9z" Nov 03 21:37:38.302078 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.302056 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f799a1-8427-44a3-a495-74720e82292a-tls-certs\") pod \"model-serving-api-75bd89fdfc-vppdx\" (UID: \"a8f799a1-8427-44a3-a495-74720e82292a\") " pod="opendatahub/model-serving-api-75bd89fdfc-vppdx" Nov 03 21:37:38.304435 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.304406 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/416c8af4-2b21-4015-9cda-e4a7f75d8506-cert\") pod \"kserve-controller-manager-776cc66d47-lrr9z\" (UID: \"416c8af4-2b21-4015-9cda-e4a7f75d8506\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lrr9z" Nov 03 21:37:38.312364 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.312332 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-2cctc"] Nov 03 21:37:38.317305 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.317270 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-ktjkl\"" Nov 03 21:37:38.317722 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.317322 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 21:37:38.317722 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.317414 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 21:37:38.317722 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.317488 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 21:37:38.317722 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.317566 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5v5cf\"" Nov 03 21:37:38.326452 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.326420 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpf7x\" (UniqueName: \"kubernetes.io/projected/416c8af4-2b21-4015-9cda-e4a7f75d8506-kube-api-access-xpf7x\") pod \"kserve-controller-manager-776cc66d47-lrr9z\" (UID: \"416c8af4-2b21-4015-9cda-e4a7f75d8506\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lrr9z" Nov 03 21:37:38.402729 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.402651 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8116c434-849d-4d12-8d67-9278246bb78b-metrics-certs\") pod \"odh-model-controller-769875db56-2cctc\" (UID: \"8116c434-849d-4d12-8d67-9278246bb78b\") " pod="opendatahub/odh-model-controller-769875db56-2cctc" Nov 03 21:37:38.402931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.402779 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvbh7\" (UniqueName: \"kubernetes.io/projected/a8f799a1-8427-44a3-a495-74720e82292a-kube-api-access-rvbh7\") pod \"model-serving-api-75bd89fdfc-vppdx\" (UID: \"a8f799a1-8427-44a3-a495-74720e82292a\") " pod="opendatahub/model-serving-api-75bd89fdfc-vppdx" Nov 03 21:37:38.402931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.402828 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2xkk\" (UniqueName: \"kubernetes.io/projected/8116c434-849d-4d12-8d67-9278246bb78b-kube-api-access-s2xkk\") pod \"odh-model-controller-769875db56-2cctc\" (UID: \"8116c434-849d-4d12-8d67-9278246bb78b\") " pod="opendatahub/odh-model-controller-769875db56-2cctc" Nov 03 21:37:38.402931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.402862 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8116c434-849d-4d12-8d67-9278246bb78b-cert\") pod \"odh-model-controller-769875db56-2cctc\" (UID: \"8116c434-849d-4d12-8d67-9278246bb78b\") " pod="opendatahub/odh-model-controller-769875db56-2cctc" Nov 03 21:37:38.402931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.402894 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f799a1-8427-44a3-a495-74720e82292a-tls-certs\") pod \"model-serving-api-75bd89fdfc-vppdx\" (UID: \"a8f799a1-8427-44a3-a495-74720e82292a\") " pod="opendatahub/model-serving-api-75bd89fdfc-vppdx" Nov 03 21:37:38.405351 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.405328 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f799a1-8427-44a3-a495-74720e82292a-tls-certs\") pod \"model-serving-api-75bd89fdfc-vppdx\" (UID: \"a8f799a1-8427-44a3-a495-74720e82292a\") " pod="opendatahub/model-serving-api-75bd89fdfc-vppdx" Nov 03 21:37:38.431150 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.431074 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvbh7\" (UniqueName: \"kubernetes.io/projected/a8f799a1-8427-44a3-a495-74720e82292a-kube-api-access-rvbh7\") pod \"model-serving-api-75bd89fdfc-vppdx\" (UID: \"a8f799a1-8427-44a3-a495-74720e82292a\") " pod="opendatahub/model-serving-api-75bd89fdfc-vppdx" Nov 03 21:37:38.503873 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.503838 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2xkk\" (UniqueName: \"kubernetes.io/projected/8116c434-849d-4d12-8d67-9278246bb78b-kube-api-access-s2xkk\") pod \"odh-model-controller-769875db56-2cctc\" (UID: \"8116c434-849d-4d12-8d67-9278246bb78b\") " pod="opendatahub/odh-model-controller-769875db56-2cctc" Nov 03 21:37:38.504045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.503897 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8116c434-849d-4d12-8d67-9278246bb78b-cert\") pod \"odh-model-controller-769875db56-2cctc\" (UID: \"8116c434-849d-4d12-8d67-9278246bb78b\") " pod="opendatahub/odh-model-controller-769875db56-2cctc" Nov 03 21:37:38.504045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.504014 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8116c434-849d-4d12-8d67-9278246bb78b-metrics-certs\") pod \"odh-model-controller-769875db56-2cctc\" (UID: \"8116c434-849d-4d12-8d67-9278246bb78b\") " pod="opendatahub/odh-model-controller-769875db56-2cctc" Nov 03 21:37:38.506523 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.506487 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8116c434-849d-4d12-8d67-9278246bb78b-cert\") pod \"odh-model-controller-769875db56-2cctc\" (UID: \"8116c434-849d-4d12-8d67-9278246bb78b\") " pod="opendatahub/odh-model-controller-769875db56-2cctc" Nov 03 21:37:38.506651 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.506543 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8116c434-849d-4d12-8d67-9278246bb78b-metrics-certs\") pod \"odh-model-controller-769875db56-2cctc\" (UID: \"8116c434-849d-4d12-8d67-9278246bb78b\") " pod="opendatahub/odh-model-controller-769875db56-2cctc" Nov 03 21:37:38.509331 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.509317 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-lrr9z" Nov 03 21:37:38.525873 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.525845 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2xkk\" (UniqueName: \"kubernetes.io/projected/8116c434-849d-4d12-8d67-9278246bb78b-kube-api-access-s2xkk\") pod \"odh-model-controller-769875db56-2cctc\" (UID: \"8116c434-849d-4d12-8d67-9278246bb78b\") " pod="opendatahub/odh-model-controller-769875db56-2cctc" Nov 03 21:37:38.564390 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.564115 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-vppdx" Nov 03 21:37:38.607905 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.607863 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-2cctc" Nov 03 21:37:38.804974 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.804947 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lrr9z"] Nov 03 21:37:38.806753 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:37:38.806724 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416c8af4_2b21_4015_9cda_e4a7f75d8506.slice/crio-9d6f0d4ad89f2e826a4e202c4b5341e9e99e25a7d5f150166ef4266d85db2578 WatchSource:0}: Error finding container 9d6f0d4ad89f2e826a4e202c4b5341e9e99e25a7d5f150166ef4266d85db2578: Status 404 returned error can't find the container with id 9d6f0d4ad89f2e826a4e202c4b5341e9e99e25a7d5f150166ef4266d85db2578 Nov 03 21:37:38.871726 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.871702 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vppdx"] Nov 03 21:37:38.873098 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:37:38.873073 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f799a1_8427_44a3_a495_74720e82292a.slice/crio-91c231ae5feb0158dc6e5b36893a6499693cfd69af6e6fcb6a864a036ab4140e WatchSource:0}: Error finding container 91c231ae5feb0158dc6e5b36893a6499693cfd69af6e6fcb6a864a036ab4140e: Status 404 returned error can't find the container with id 91c231ae5feb0158dc6e5b36893a6499693cfd69af6e6fcb6a864a036ab4140e Nov 03 21:37:38.915177 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:38.915151 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-2cctc"] Nov 03 21:37:38.917223 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:37:38.917192 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8116c434_849d_4d12_8d67_9278246bb78b.slice/crio-e1ba0fd8b23dbff97cdcaa775b4593744569d4244ec334268a7dcf09c60504f2 WatchSource:0}: Error finding container e1ba0fd8b23dbff97cdcaa775b4593744569d4244ec334268a7dcf09c60504f2: Status 404 returned error can't find the container with id e1ba0fd8b23dbff97cdcaa775b4593744569d4244ec334268a7dcf09c60504f2 Nov 03 21:37:39.191327 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:39.191222 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-2cctc" event={"ID":"8116c434-849d-4d12-8d67-9278246bb78b","Type":"ContainerStarted","Data":"e1ba0fd8b23dbff97cdcaa775b4593744569d4244ec334268a7dcf09c60504f2"} Nov 03 21:37:39.192614 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:39.192588 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-vppdx" event={"ID":"a8f799a1-8427-44a3-a495-74720e82292a","Type":"ContainerStarted","Data":"91c231ae5feb0158dc6e5b36893a6499693cfd69af6e6fcb6a864a036ab4140e"} Nov 03 21:37:39.193852 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:39.193830 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lrr9z" event={"ID":"416c8af4-2b21-4015-9cda-e4a7f75d8506","Type":"ContainerStarted","Data":"9d6f0d4ad89f2e826a4e202c4b5341e9e99e25a7d5f150166ef4266d85db2578"} Nov 03 21:37:40.787320 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:37:40.787280 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:37:41.218088 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:41.218046 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-vppdx" event={"ID":"a8f799a1-8427-44a3-a495-74720e82292a","Type":"ContainerStarted","Data":"cb556b268d88e8d0509c2fab569984b6c3eeb2bee18839013f4c0e6bf4f97026"} Nov 03 21:37:41.218348 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:41.218327 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-vppdx" Nov 03 21:37:41.261694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:41.261625 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-vppdx" podStartSLOduration=1.089826396 podStartE2EDuration="3.261610434s" totalImagesPullingTime="2.171784038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:41.260236416 +0000 UTC m=+476.767464761" watchObservedRunningTime="2026-11-03 21:37:41.261610434 +0000 UTC m=+476.768838714" Nov 03 21:37:42.227136 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:42.227098 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-2cctc" event={"ID":"8116c434-849d-4d12-8d67-9278246bb78b","Type":"ContainerStarted","Data":"d034f7fe8b65318565803b28d49749bcfc5f8b4e3ba158b032cf925b52e26536"} Nov 03 21:37:42.227607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:42.227231 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-2cctc" Nov 03 21:37:42.229037 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:42.229013 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lrr9z" event={"ID":"416c8af4-2b21-4015-9cda-e4a7f75d8506","Type":"ContainerStarted","Data":"2e10e296ee292213977a646559d820796fc641c3afac38a2c47b2d0511ee356d"} Nov 03 21:37:42.229192 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:42.229102 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lrr9z" Nov 03 21:37:42.269100 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:42.269041 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-2cctc" podStartSLOduration=1.153726014 podStartE2EDuration="4.269027575s" totalImagesPullingTime="3.115301561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:42.26762182 +0000 UTC m=+477.774850120" watchObservedRunningTime="2026-11-03 21:37:42.269027575 +0000 UTC m=+477.776255875" Nov 03 21:37:42.311121 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:42.310993 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-lrr9z" podStartSLOduration=1.50152966 podStartE2EDuration="4.310973689s" totalImagesPullingTime="2.809444029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:42.308102454 +0000 UTC m=+477.815330736" watchObservedRunningTime="2026-11-03 21:37:42.310973689 +0000 UTC m=+477.818201969" Nov 03 21:37:43.161544 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:43.161512 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-95xs7" Nov 03 21:37:52.234354 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:52.234324 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-vppdx" Nov 03 21:37:53.238405 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:37:53.238362 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-2cctc" Nov 03 21:38:00.726568 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:00.726528 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh"] Nov 03 21:38:00.801612 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:00.801562 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh"] Nov 03 21:38:00.801797 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:00.801701 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:00.938226 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:00.938187 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44bc42a8-430e-44ec-b717-d1a2436968f5-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:00.938226 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:00.938227 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44bc42a8-430e-44ec-b717-d1a2436968f5-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:00.938449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:00.938248 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44bc42a8-430e-44ec-b717-d1a2436968f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:00.938449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:00.938325 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44bc42a8-430e-44ec-b717-d1a2436968f5-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:00.938449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:00.938379 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkj49\" (UniqueName: \"kubernetes.io/projected/44bc42a8-430e-44ec-b717-d1a2436968f5-kube-api-access-mkj49\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:00.938449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:00.938414 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44bc42a8-430e-44ec-b717-d1a2436968f5-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:00.938589 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:00.938517 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44bc42a8-430e-44ec-b717-d1a2436968f5-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:00.938589 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:00.938560 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44bc42a8-430e-44ec-b717-d1a2436968f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:00.938690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:00.938655 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44bc42a8-430e-44ec-b717-d1a2436968f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.039476 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.039382 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44bc42a8-430e-44ec-b717-d1a2436968f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.039639 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.039544 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44bc42a8-430e-44ec-b717-d1a2436968f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.039639 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.039633 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44bc42a8-430e-44ec-b717-d1a2436968f5-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.039778 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.039707 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44bc42a8-430e-44ec-b717-d1a2436968f5-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.039778 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.039734 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44bc42a8-430e-44ec-b717-d1a2436968f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.039778 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.039765 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44bc42a8-430e-44ec-b717-d1a2436968f5-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.039936 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.039800 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkj49\" (UniqueName: \"kubernetes.io/projected/44bc42a8-430e-44ec-b717-d1a2436968f5-kube-api-access-mkj49\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.039936 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.039832 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44bc42a8-430e-44ec-b717-d1a2436968f5-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.039936 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.039880 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44bc42a8-430e-44ec-b717-d1a2436968f5-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.040149 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.040117 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44bc42a8-430e-44ec-b717-d1a2436968f5-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.040328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.040254 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44bc42a8-430e-44ec-b717-d1a2436968f5-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.040545 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.040521 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44bc42a8-430e-44ec-b717-d1a2436968f5-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.041366 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.040765 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44bc42a8-430e-44ec-b717-d1a2436968f5-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.041366 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.040838 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44bc42a8-430e-44ec-b717-d1a2436968f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.042606 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.042581 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44bc42a8-430e-44ec-b717-d1a2436968f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.042859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.042836 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44bc42a8-430e-44ec-b717-d1a2436968f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.060502 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.060477 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkj49\" (UniqueName: \"kubernetes.io/projected/44bc42a8-430e-44ec-b717-d1a2436968f5-kube-api-access-mkj49\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.060620 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.060584 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44bc42a8-430e-44ec-b717-d1a2436968f5-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-wc8rh\" (UID: \"44bc42a8-430e-44ec-b717-d1a2436968f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.130499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.130463 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:01.453653 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.453624 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh"] Nov 03 21:38:01.454994 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:38:01.454966 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44bc42a8_430e_44ec_b717_d1a2436968f5.slice/crio-e0f867bfa034695280d99e4cb42b6038c0c8d223c2d93bd43384fb017b085f59 WatchSource:0}: Error finding container e0f867bfa034695280d99e4cb42b6038c0c8d223c2d93bd43384fb017b085f59: Status 404 returned error can't find the container with id e0f867bfa034695280d99e4cb42b6038c0c8d223c2d93bd43384fb017b085f59 Nov 03 21:38:01.457348 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.457317 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:38:01.457430 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.457377 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:38:01.457430 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:01.457409 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:38:02.371456 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:02.371425 5041 generic.go:356] "Generic (PLEG): container finished" podID="bd662fa8-196b-4cad-b414-1bc2dbd68aa1" containerID="c7f3f2d62c1a8797934578723ff3a60a022c5fc9bda6e454cbbd61a2800f4047" exitCode=0 Nov 03 21:38:02.371937 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:02.371496 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" event={"ID":"bd662fa8-196b-4cad-b414-1bc2dbd68aa1","Type":"ContainerDied","Data":"c7f3f2d62c1a8797934578723ff3a60a022c5fc9bda6e454cbbd61a2800f4047"} Nov 03 21:38:02.371937 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:02.371928 5041 scope.go:122] "RemoveContainer" containerID="c7f3f2d62c1a8797934578723ff3a60a022c5fc9bda6e454cbbd61a2800f4047" Nov 03 21:38:02.373212 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:02.373184 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:38:02.373334 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:02.373266 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:38:02.373334 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:02.373314 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:38:02.373449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:02.373348 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" event={"ID":"44bc42a8-430e-44ec-b717-d1a2436968f5","Type":"ContainerStarted","Data":"8e31fc2f64ee83b1969398dd0db7c71264bb9a814c3af5bee3ddb59e608b478c"} Nov 03 21:38:02.373449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:02.373395 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" event={"ID":"44bc42a8-430e-44ec-b717-d1a2436968f5","Type":"ContainerStarted","Data":"e0f867bfa034695280d99e4cb42b6038c0c8d223c2d93bd43384fb017b085f59"} Nov 03 21:38:02.478515 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:02.477154 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" podStartSLOduration=2.477134346 podStartE2EDuration="2.477134346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:02.473983881 +0000 UTC m=+497.981212182" watchObservedRunningTime="2026-11-03 21:38:02.477134346 +0000 UTC m=+497.984362629" Nov 03 21:38:03.131685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:03.131058 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:03.136687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:03.134593 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-wc8rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 21:38:03.136687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:03.134679 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" podUID="44bc42a8-430e-44ec-b717-d1a2436968f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 21:38:03.380686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:03.380564 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" event={"ID":"bd662fa8-196b-4cad-b414-1bc2dbd68aa1","Type":"ContainerStarted","Data":"6534249a10d9dcc504e8a5697ff5b939d3337477ec0800fd77c4f8204af05ed9"} Nov 03 21:38:03.540684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:03.539883 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:38:03.540684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:03.540015 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6kgrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 03 21:38:03.540684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:03.540057 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" podUID="bd662fa8-196b-4cad-b414-1bc2dbd68aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 03 21:38:04.131687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:04.131101 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-wc8rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 21:38:04.131687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:04.131174 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" podUID="44bc42a8-430e-44ec-b717-d1a2436968f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 21:38:04.549686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:04.541083 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6kgrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 03 21:38:04.549686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:04.541188 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" podUID="bd662fa8-196b-4cad-b414-1bc2dbd68aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 03 21:38:05.134685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:05.134094 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-wc8rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 21:38:05.134685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:05.134179 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" podUID="44bc42a8-430e-44ec-b717-d1a2436968f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 21:38:05.540172 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:05.540134 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6kgrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 03 21:38:05.540487 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:05.540452 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" podUID="bd662fa8-196b-4cad-b414-1bc2dbd68aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 03 21:38:06.136692 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:06.132034 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-wc8rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 21:38:06.136692 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:06.132112 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" podUID="44bc42a8-430e-44ec-b717-d1a2436968f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 21:38:06.541550 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:06.541065 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6kgrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 03 21:38:06.541550 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:06.541147 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" podUID="bd662fa8-196b-4cad-b414-1bc2dbd68aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 03 21:38:07.132107 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:07.132072 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-wc8rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 21:38:07.132379 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:07.132350 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" podUID="44bc42a8-430e-44ec-b717-d1a2436968f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 21:38:07.539124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:07.539074 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6kgrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 03 21:38:07.539537 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:07.539142 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" podUID="bd662fa8-196b-4cad-b414-1bc2dbd68aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 03 21:38:08.136817 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:08.136782 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:08.137095 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:08.137072 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:08.137796 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:08.137775 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" Nov 03 21:38:08.542093 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:08.542063 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:38:08.542547 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:08.542525 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:38:08.543149 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:08.543131 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" Nov 03 21:38:10.777281 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:38:10.777244 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:38:13.241589 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:13.241556 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lrr9z" Nov 03 21:38:40.778348 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:38:40.778303 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:38:45.865993 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.865957 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr"] Nov 03 21:38:45.869612 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.869591 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.874100 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.873995 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69cc2bab-a8e9-4a4a-a114-88663140640d-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.874100 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.874075 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69cc2bab-a8e9-4a4a-a114-88663140640d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.874307 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.874140 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cc2bab-a8e9-4a4a-a114-88663140640d-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.874307 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.874165 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69cc2bab-a8e9-4a4a-a114-88663140640d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.874307 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.874192 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69cc2bab-a8e9-4a4a-a114-88663140640d-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.874307 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.874240 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69cc2bab-a8e9-4a4a-a114-88663140640d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.874519 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.874494 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cc2bab-a8e9-4a4a-a114-88663140640d-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.874643 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.874616 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69cc2bab-a8e9-4a4a-a114-88663140640d-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.891103 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.891069 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr"] Nov 03 21:38:45.975997 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.975955 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69cc2bab-a8e9-4a4a-a114-88663140640d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.976205 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.976022 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cc2bab-a8e9-4a4a-a114-88663140640d-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.976205 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.976045 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69cc2bab-a8e9-4a4a-a114-88663140640d-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.976205 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.976071 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwhht\" (UniqueName: \"kubernetes.io/projected/69cc2bab-a8e9-4a4a-a114-88663140640d-kube-api-access-nwhht\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.976205 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.976119 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69cc2bab-a8e9-4a4a-a114-88663140640d-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.976205 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.976154 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69cc2bab-a8e9-4a4a-a114-88663140640d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.976205 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.976183 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cc2bab-a8e9-4a4a-a114-88663140640d-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.976521 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.976249 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69cc2bab-a8e9-4a4a-a114-88663140640d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.976521 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.976300 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69cc2bab-a8e9-4a4a-a114-88663140640d-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.976639 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.976526 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cc2bab-a8e9-4a4a-a114-88663140640d-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.976639 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.976584 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69cc2bab-a8e9-4a4a-a114-88663140640d-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.976806 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.976700 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cc2bab-a8e9-4a4a-a114-88663140640d-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.976806 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.976720 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69cc2bab-a8e9-4a4a-a114-88663140640d-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.977213 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.977188 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69cc2bab-a8e9-4a4a-a114-88663140640d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.978514 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.978487 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69cc2bab-a8e9-4a4a-a114-88663140640d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.978768 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.978748 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69cc2bab-a8e9-4a4a-a114-88663140640d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:45.996538 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:45.996496 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69cc2bab-a8e9-4a4a-a114-88663140640d-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:46.077262 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:46.077197 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwhht\" (UniqueName: \"kubernetes.io/projected/69cc2bab-a8e9-4a4a-a114-88663140640d-kube-api-access-nwhht\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:46.098935 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:46.098895 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwhht\" (UniqueName: \"kubernetes.io/projected/69cc2bab-a8e9-4a4a-a114-88663140640d-kube-api-access-nwhht\") pod \"maas-default-gateway-openshift-default-75d7c785f-8b9jr\" (UID: \"69cc2bab-a8e9-4a4a-a114-88663140640d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:46.185984 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:46.185885 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:46.496812 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:46.496779 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr"] Nov 03 21:38:46.498632 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:38:46.498598 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cc2bab_a8e9_4a4a_a114_88663140640d.slice/crio-678b7c7626dc680f0b02b737cfcf759ae07ff764913b6867417d33d3281451ce WatchSource:0}: Error finding container 678b7c7626dc680f0b02b737cfcf759ae07ff764913b6867417d33d3281451ce: Status 404 returned error can't find the container with id 678b7c7626dc680f0b02b737cfcf759ae07ff764913b6867417d33d3281451ce Nov 03 21:38:46.500956 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:46.500924 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:38:46.501018 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:46.500990 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:38:46.501053 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:46.501020 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:38:46.635979 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:46.635932 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" event={"ID":"69cc2bab-a8e9-4a4a-a114-88663140640d","Type":"ContainerStarted","Data":"6496a53fe464f3e79bff98ac1c8b9fe27e629e2130babb01a1ec5ec2e4b52e17"} Nov 03 21:38:46.636162 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:46.636009 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" event={"ID":"69cc2bab-a8e9-4a4a-a114-88663140640d","Type":"ContainerStarted","Data":"678b7c7626dc680f0b02b737cfcf759ae07ff764913b6867417d33d3281451ce"} Nov 03 21:38:46.679795 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:46.679733 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" podStartSLOduration=1.6797173619999999 podStartE2EDuration="1.679717362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:46.678343649 +0000 UTC m=+542.185571929" watchObservedRunningTime="2026-11-03 21:38:46.679717362 +0000 UTC m=+542.186945642" Nov 03 21:38:47.186856 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:47.186811 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:47.188479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:47.188455 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-8b9jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 21:38:47.188543 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:47.188515 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" podUID="69cc2bab-a8e9-4a4a-a114-88663140640d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 21:38:48.187314 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:48.187272 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-8b9jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 21:38:48.187753 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:48.187334 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" podUID="69cc2bab-a8e9-4a4a-a114-88663140640d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 21:38:49.186651 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:49.186613 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-8b9jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 21:38:49.186861 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:49.186702 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" podUID="69cc2bab-a8e9-4a4a-a114-88663140640d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 21:38:50.186358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:50.186318 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-8b9jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 21:38:50.186807 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:50.186396 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" podUID="69cc2bab-a8e9-4a4a-a114-88663140640d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 21:38:51.186527 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:51.186488 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-8b9jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 21:38:51.186966 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:51.186547 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" podUID="69cc2bab-a8e9-4a4a-a114-88663140640d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 21:38:52.186581 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:52.186541 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-8b9jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 21:38:52.187021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:52.186601 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" podUID="69cc2bab-a8e9-4a4a-a114-88663140640d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 21:38:53.190523 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:53.190486 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:53.190938 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:53.190778 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:38:53.191535 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:38:53.191513 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" Nov 03 21:39:10.779084 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:39:10.779038 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:39:28.174123 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.174038 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk"] Nov 03 21:39:28.177182 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.177163 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.199764 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.199734 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk"] Nov 03 21:39:28.199932 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.199902 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f3d92ef-2874-44da-90a5-ff115f8c2755-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.199992 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.199976 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f3d92ef-2874-44da-90a5-ff115f8c2755-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.200031 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.200008 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f3d92ef-2874-44da-90a5-ff115f8c2755-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.200071 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.200060 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f3d92ef-2874-44da-90a5-ff115f8c2755-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.200111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.200100 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f3d92ef-2874-44da-90a5-ff115f8c2755-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.200151 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.200132 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxc28\" (UniqueName: \"kubernetes.io/projected/2f3d92ef-2874-44da-90a5-ff115f8c2755-kube-api-access-bxc28\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.210936 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.210885 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 21:39:28.210936 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.210912 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 21:39:28.211168 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.211065 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 21:39:28.211168 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.211150 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xpfzh\"" Nov 03 21:39:28.300620 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.300585 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f3d92ef-2874-44da-90a5-ff115f8c2755-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.300620 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.300622 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f3d92ef-2874-44da-90a5-ff115f8c2755-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.300896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.300675 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f3d92ef-2874-44da-90a5-ff115f8c2755-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.300896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.300704 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f3d92ef-2874-44da-90a5-ff115f8c2755-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.300896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.300723 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxc28\" (UniqueName: \"kubernetes.io/projected/2f3d92ef-2874-44da-90a5-ff115f8c2755-kube-api-access-bxc28\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.300896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.300782 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f3d92ef-2874-44da-90a5-ff115f8c2755-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.301134 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.301076 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f3d92ef-2874-44da-90a5-ff115f8c2755-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.301134 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.301126 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f3d92ef-2874-44da-90a5-ff115f8c2755-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.301222 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.301199 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f3d92ef-2874-44da-90a5-ff115f8c2755-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.303044 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.303028 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f3d92ef-2874-44da-90a5-ff115f8c2755-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.303241 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.303226 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f3d92ef-2874-44da-90a5-ff115f8c2755-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.320785 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.320755 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxc28\" (UniqueName: \"kubernetes.io/projected/2f3d92ef-2874-44da-90a5-ff115f8c2755-kube-api-access-bxc28\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bqstk\" (UID: \"2f3d92ef-2874-44da-90a5-ff115f8c2755\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.492593 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.492550 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:28.792544 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.792518 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk"] Nov 03 21:39:28.794269 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:39:28.794231 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3d92ef_2874_44da_90a5_ff115f8c2755.slice/crio-303dd593d1f92ed0df6075f49de5052c9cbfe02786c00bcc0d7d31526d2978af WatchSource:0}: Error finding container 303dd593d1f92ed0df6075f49de5052c9cbfe02786c00bcc0d7d31526d2978af: Status 404 returned error can't find the container with id 303dd593d1f92ed0df6075f49de5052c9cbfe02786c00bcc0d7d31526d2978af Nov 03 21:39:28.866548 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:28.866507 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" event={"ID":"2f3d92ef-2874-44da-90a5-ff115f8c2755","Type":"ContainerStarted","Data":"303dd593d1f92ed0df6075f49de5052c9cbfe02786c00bcc0d7d31526d2978af"} Nov 03 21:39:30.901692 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:30.901640 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" event={"ID":"2f3d92ef-2874-44da-90a5-ff115f8c2755","Type":"ContainerStarted","Data":"8fd7015753d34d8539b51fd76481b0decd8ff9c41d5919a1368c381dca1629ad"} Nov 03 21:39:30.945462 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:30.945408 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" podStartSLOduration=1.329767087 podStartE2EDuration="2.94539436s" totalImagesPullingTime="1.615627273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:30.943323523 +0000 UTC m=+586.450551803" watchObservedRunningTime="2026-11-03 21:39:30.94539436 +0000 UTC m=+586.452622639" Nov 03 21:39:31.493136 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:31.493103 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:33.466250 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.466217 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb"] Nov 03 21:39:33.469182 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.469160 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.485142 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.485118 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 21:39:33.501268 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.501239 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb"] Nov 03 21:39:33.567153 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.567119 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa892a3f-4760-45ff-b00b-0840970540bc-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.567337 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.567163 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa892a3f-4760-45ff-b00b-0840970540bc-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.567337 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.567257 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa892a3f-4760-45ff-b00b-0840970540bc-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.567337 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.567328 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa892a3f-4760-45ff-b00b-0840970540bc-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.567456 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.567400 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9jwr\" (UniqueName: \"kubernetes.io/projected/fa892a3f-4760-45ff-b00b-0840970540bc-kube-api-access-q9jwr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.567456 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.567451 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa892a3f-4760-45ff-b00b-0840970540bc-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.668812 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.668777 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa892a3f-4760-45ff-b00b-0840970540bc-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.669004 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.668826 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9jwr\" (UniqueName: \"kubernetes.io/projected/fa892a3f-4760-45ff-b00b-0840970540bc-kube-api-access-q9jwr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.669004 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.668867 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa892a3f-4760-45ff-b00b-0840970540bc-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.669004 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.668896 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa892a3f-4760-45ff-b00b-0840970540bc-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.669004 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.668930 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa892a3f-4760-45ff-b00b-0840970540bc-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.669212 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.669010 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa892a3f-4760-45ff-b00b-0840970540bc-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.669400 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.669376 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa892a3f-4760-45ff-b00b-0840970540bc-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.669476 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.669444 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa892a3f-4760-45ff-b00b-0840970540bc-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.669476 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.669428 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa892a3f-4760-45ff-b00b-0840970540bc-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.671355 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.671330 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa892a3f-4760-45ff-b00b-0840970540bc-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.671419 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.671382 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa892a3f-4760-45ff-b00b-0840970540bc-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.689376 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.689349 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9jwr\" (UniqueName: \"kubernetes.io/projected/fa892a3f-4760-45ff-b00b-0840970540bc-kube-api-access-q9jwr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-v2pwb\" (UID: \"fa892a3f-4760-45ff-b00b-0840970540bc\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:33.784373 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:33.784288 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:34.081236 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:34.078643 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb"] Nov 03 21:39:34.084339 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:39:34.084304 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa892a3f_4760_45ff_b00b_0840970540bc.slice/crio-a5d2a25aa9d457a3c3e471dd46a28ccbb184178fb83b1f8c0296f2ae87618763 WatchSource:0}: Error finding container a5d2a25aa9d457a3c3e471dd46a28ccbb184178fb83b1f8c0296f2ae87618763: Status 404 returned error can't find the container with id a5d2a25aa9d457a3c3e471dd46a28ccbb184178fb83b1f8c0296f2ae87618763 Nov 03 21:39:34.928940 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:34.928902 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" event={"ID":"fa892a3f-4760-45ff-b00b-0840970540bc","Type":"ContainerStarted","Data":"164952aa6fe1ef2e4046cd35313584c02054f812164413500a84f2cdc2963c6c"} Nov 03 21:39:34.929361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:34.928972 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" event={"ID":"fa892a3f-4760-45ff-b00b-0840970540bc","Type":"ContainerStarted","Data":"a5d2a25aa9d457a3c3e471dd46a28ccbb184178fb83b1f8c0296f2ae87618763"} Nov 03 21:39:34.974281 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:34.974216 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" podStartSLOduration=1.7871280920000001 podStartE2EDuration="1.974196219s" totalImagesPullingTime="187.068127ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:34.972086234 +0000 UTC m=+590.479314515" watchObservedRunningTime="2026-11-03 21:39:34.974196219 +0000 UTC m=+590.481424500" Nov 03 21:39:35.788824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:35.785772 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:37.477950 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.477914 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r"] Nov 03 21:39:37.481321 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.481291 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.503565 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.503532 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r"] Nov 03 21:39:37.513432 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.513406 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 21:39:37.516317 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.516292 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xbxp\" (UniqueName: \"kubernetes.io/projected/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-kube-api-access-5xbxp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.516452 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.516340 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.516452 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.516410 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.516452 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.516438 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.516623 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.516460 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.516623 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.516481 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.617774 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.617720 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xbxp\" (UniqueName: \"kubernetes.io/projected/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-kube-api-access-5xbxp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.617951 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.617817 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.617951 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.617847 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.617951 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.617882 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.617951 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.617919 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.618189 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.617953 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.618355 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.618324 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.618548 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.618375 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.618548 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.618413 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.620470 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.620444 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.620711 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.620692 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.638954 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.638925 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xbxp\" (UniqueName: \"kubernetes.io/projected/1d7a34c6-9da5-45de-9373-50b3b18eb7b1-kube-api-access-5xbxp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r\" (UID: \"1d7a34c6-9da5-45de-9373-50b3b18eb7b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:37.795118 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:37.795001 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:38.124554 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:38.124461 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r"] Nov 03 21:39:38.129179 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:39:38.129141 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7a34c6_9da5_45de_9373_50b3b18eb7b1.slice/crio-83971cfffed36594f23d623ae1c8f4b41f88918b73a7dc6cd61afa6f689e3941 WatchSource:0}: Error finding container 83971cfffed36594f23d623ae1c8f4b41f88918b73a7dc6cd61afa6f689e3941: Status 404 returned error can't find the container with id 83971cfffed36594f23d623ae1c8f4b41f88918b73a7dc6cd61afa6f689e3941 Nov 03 21:39:38.495841 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:38.494647 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:38.529713 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:38.519765 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:39.075297 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:39.074734 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" event={"ID":"1d7a34c6-9da5-45de-9373-50b3b18eb7b1","Type":"ContainerStarted","Data":"3c1d8dd1fbc7a40e89011804d2582ad30035456a21d212ed2954179ee3e5efc9"} Nov 03 21:39:39.075297 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:39.074813 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" event={"ID":"1d7a34c6-9da5-45de-9373-50b3b18eb7b1","Type":"ContainerStarted","Data":"83971cfffed36594f23d623ae1c8f4b41f88918b73a7dc6cd61afa6f689e3941"} Nov 03 21:39:39.094653 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:39.094621 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" Nov 03 21:39:39.223514 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:39.223441 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" podStartSLOduration=1.980983963 podStartE2EDuration="2.223418428s" totalImagesPullingTime="242.434465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:39.132500898 +0000 UTC m=+594.639729180" watchObservedRunningTime="2026-11-03 21:39:39.223418428 +0000 UTC m=+594.730646708" Nov 03 21:39:39.795500 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:39.795460 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:40.803084 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:39:40.803045 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:39:42.837684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:42.837633 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db"] Nov 03 21:39:42.897691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:42.890444 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db"] Nov 03 21:39:42.897691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:42.890578 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:42.977686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:42.958239 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fe7207f-2102-4312-b28d-9c91c22b0ddb-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:42.998342 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:42.998308 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fe7207f-2102-4312-b28d-9c91c22b0ddb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:42.998744 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:42.998715 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fe7207f-2102-4312-b28d-9c91c22b0ddb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:42.999051 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:42.999031 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fe7207f-2102-4312-b28d-9c91c22b0ddb-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:42.999193 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:42.999173 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46l8q\" (UniqueName: \"kubernetes.io/projected/6fe7207f-2102-4312-b28d-9c91c22b0ddb-kube-api-access-46l8q\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:42.999345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:42.999328 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fe7207f-2102-4312-b28d-9c91c22b0ddb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:42.999536 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:42.998069 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 21:39:43.102457 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:43.100211 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fe7207f-2102-4312-b28d-9c91c22b0ddb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:43.102457 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:43.100330 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fe7207f-2102-4312-b28d-9c91c22b0ddb-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:43.102457 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:43.100363 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46l8q\" (UniqueName: \"kubernetes.io/projected/6fe7207f-2102-4312-b28d-9c91c22b0ddb-kube-api-access-46l8q\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:43.102457 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:43.100414 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fe7207f-2102-4312-b28d-9c91c22b0ddb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:43.102457 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:43.100444 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fe7207f-2102-4312-b28d-9c91c22b0ddb-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:43.102457 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:43.100489 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fe7207f-2102-4312-b28d-9c91c22b0ddb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:43.102457 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:43.101085 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fe7207f-2102-4312-b28d-9c91c22b0ddb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:43.102457 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:43.102053 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fe7207f-2102-4312-b28d-9c91c22b0ddb-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:43.103732 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:43.103573 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fe7207f-2102-4312-b28d-9c91c22b0ddb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:43.122545 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:43.122463 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fe7207f-2102-4312-b28d-9c91c22b0ddb-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:43.158684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:43.146483 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46l8q\" (UniqueName: \"kubernetes.io/projected/6fe7207f-2102-4312-b28d-9c91c22b0ddb-kube-api-access-46l8q\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:43.160055 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:43.160028 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fe7207f-2102-4312-b28d-9c91c22b0ddb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db\" (UID: \"6fe7207f-2102-4312-b28d-9c91c22b0ddb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:43.260830 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:43.260025 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:43.795760 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:43.794069 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:43.795760 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:43.795559 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db"] Nov 03 21:39:43.876690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:43.847038 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:44.123430 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:44.123378 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" event={"ID":"6fe7207f-2102-4312-b28d-9c91c22b0ddb","Type":"ContainerStarted","Data":"00b0810f57b3b7e649a70c8ff97a1cbb74b5755af3a11cc229b021040b307a4e"} Nov 03 21:39:44.143107 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:44.143078 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" Nov 03 21:39:44.693598 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:44.693567 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 21:39:44.694910 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:44.694885 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 21:39:45.130512 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:45.130469 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" event={"ID":"6fe7207f-2102-4312-b28d-9c91c22b0ddb","Type":"ContainerStarted","Data":"19c3134ae8aee281ff55c524ade97bc1a0e3ec45626fb53f20abf73bb7654b7b"} Nov 03 21:39:45.176591 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:45.176510 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" podStartSLOduration=2.903066379 podStartE2EDuration="3.17648841s" totalImagesPullingTime="273.422031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:45.173738011 +0000 UTC m=+600.680966294" watchObservedRunningTime="2026-11-03 21:39:45.17648841 +0000 UTC m=+600.683716693" Nov 03 21:39:45.260682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:45.260639 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:46.020310 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.020269 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4"] Nov 03 21:39:46.023911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.023880 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.050187 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.049434 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4"] Nov 03 21:39:46.167152 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.167116 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be23a49-ab91-409d-b131-cb4a917a58e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.167152 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.167151 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be23a49-ab91-409d-b131-cb4a917a58e3-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.167800 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.167174 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be23a49-ab91-409d-b131-cb4a917a58e3-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.167800 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.167200 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be23a49-ab91-409d-b131-cb4a917a58e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.167800 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.167360 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be23a49-ab91-409d-b131-cb4a917a58e3-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.167800 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.167422 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be23a49-ab91-409d-b131-cb4a917a58e3-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.167800 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.167521 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be23a49-ab91-409d-b131-cb4a917a58e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.167800 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.167573 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2spf\" (UniqueName: \"kubernetes.io/projected/9be23a49-ab91-409d-b131-cb4a917a58e3-kube-api-access-n2spf\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.167800 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.167696 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be23a49-ab91-409d-b131-cb4a917a58e3-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.268484 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.268442 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be23a49-ab91-409d-b131-cb4a917a58e3-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.268484 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.268492 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be23a49-ab91-409d-b131-cb4a917a58e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.268820 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.268512 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be23a49-ab91-409d-b131-cb4a917a58e3-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.268820 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.268539 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be23a49-ab91-409d-b131-cb4a917a58e3-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.268820 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.268704 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be23a49-ab91-409d-b131-cb4a917a58e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.269000 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.268837 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be23a49-ab91-409d-b131-cb4a917a58e3-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.269000 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.268968 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be23a49-ab91-409d-b131-cb4a917a58e3-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.269265 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.269199 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be23a49-ab91-409d-b131-cb4a917a58e3-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.269265 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.269223 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be23a49-ab91-409d-b131-cb4a917a58e3-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.269425 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.269285 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be23a49-ab91-409d-b131-cb4a917a58e3-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.270195 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.270150 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be23a49-ab91-409d-b131-cb4a917a58e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.270299 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.270196 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be23a49-ab91-409d-b131-cb4a917a58e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.270299 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.270241 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2spf\" (UniqueName: \"kubernetes.io/projected/9be23a49-ab91-409d-b131-cb4a917a58e3-kube-api-access-n2spf\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.275881 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.270604 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be23a49-ab91-409d-b131-cb4a917a58e3-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.275881 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.272848 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be23a49-ab91-409d-b131-cb4a917a58e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.296122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.296085 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be23a49-ab91-409d-b131-cb4a917a58e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.296304 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.296230 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be23a49-ab91-409d-b131-cb4a917a58e3-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.297784 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.297754 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2spf\" (UniqueName: \"kubernetes.io/projected/9be23a49-ab91-409d-b131-cb4a917a58e3-kube-api-access-n2spf\") pod \"maas-default-gateway-openshift-default-75d7c785f-lt4j4\" (UID: \"9be23a49-ab91-409d-b131-cb4a917a58e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.340028 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.339984 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:46.700567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.700542 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4"] Nov 03 21:39:46.703405 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:39:46.703374 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be23a49_ab91_409d_b131_cb4a917a58e3.slice/crio-0657b40df23e5f7c84ab4d8381f14a12c62cd94f93ec5f80d32a4a718eb519e1 WatchSource:0}: Error finding container 0657b40df23e5f7c84ab4d8381f14a12c62cd94f93ec5f80d32a4a718eb519e1: Status 404 returned error can't find the container with id 0657b40df23e5f7c84ab4d8381f14a12c62cd94f93ec5f80d32a4a718eb519e1 Nov 03 21:39:46.706377 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.706340 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:39:46.706487 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.706411 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:39:46.706487 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:46.706441 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:39:47.192860 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:47.191835 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" event={"ID":"9be23a49-ab91-409d-b131-cb4a917a58e3","Type":"ContainerStarted","Data":"4e8ea960c7d69c266a4ea84574e3df512e1a671c21363406f5c4e35faac0f20a"} Nov 03 21:39:47.192860 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:47.191910 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" event={"ID":"9be23a49-ab91-409d-b131-cb4a917a58e3","Type":"ContainerStarted","Data":"0657b40df23e5f7c84ab4d8381f14a12c62cd94f93ec5f80d32a4a718eb519e1"} Nov 03 21:39:47.274804 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:47.265377 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" podStartSLOduration=2.265340445 podStartE2EDuration="2.265340445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:47.262307131 +0000 UTC m=+602.769535413" watchObservedRunningTime="2026-11-03 21:39:47.265340445 +0000 UTC m=+602.772568726" Nov 03 21:39:47.343687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:47.342739 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:47.347684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:47.346799 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lt4j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 21:39:47.347684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:47.346867 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" podUID="9be23a49-ab91-409d-b131-cb4a917a58e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 21:39:47.798694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:47.797736 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:47.844193 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:47.844147 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:48.248582 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:48.248276 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r" Nov 03 21:39:48.357688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:48.356760 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lt4j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 21:39:48.381687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:48.365754 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" podUID="9be23a49-ab91-409d-b131-cb4a917a58e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 21:39:49.053795 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:49.053749 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-6d575fdb56-8br4h" podUID="d82b63ac-65d4-4e30-8b51-d308cb3d335d" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.50:8081/readyz\": dial tcp 10.132.0.50:8081: connect: connection refused" Nov 03 21:39:49.230691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:49.230623 5041 generic.go:356] "Generic (PLEG): container finished" podID="d82b63ac-65d4-4e30-8b51-d308cb3d335d" containerID="e6f611effcbffbd5ced79fc870a015b984cc5d508dc3259d208460ee36d03206" exitCode=1 Nov 03 21:39:49.232686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:49.231345 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d575fdb56-8br4h" event={"ID":"d82b63ac-65d4-4e30-8b51-d308cb3d335d","Type":"ContainerDied","Data":"e6f611effcbffbd5ced79fc870a015b984cc5d508dc3259d208460ee36d03206"} Nov 03 21:39:49.236043 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:49.236012 5041 scope.go:122] "RemoveContainer" containerID="e6f611effcbffbd5ced79fc870a015b984cc5d508dc3259d208460ee36d03206" Nov 03 21:39:49.341387 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:49.341303 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lt4j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 21:39:49.341387 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:49.341370 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" podUID="9be23a49-ab91-409d-b131-cb4a917a58e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 21:39:50.248591 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:50.248546 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d575fdb56-8br4h" event={"ID":"d82b63ac-65d4-4e30-8b51-d308cb3d335d","Type":"ContainerStarted","Data":"3d0d9132064b1cd62436fc80c5bc75111a1ff534730e9f63b17321577836678d"} Nov 03 21:39:50.248873 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:50.248794 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:39:50.341272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:50.341203 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lt4j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 21:39:50.341476 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:50.341291 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" podUID="9be23a49-ab91-409d-b131-cb4a917a58e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 21:39:51.341285 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:51.341244 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lt4j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 21:39:51.341470 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:51.341301 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" podUID="9be23a49-ab91-409d-b131-cb4a917a58e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 21:39:52.341251 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:52.341210 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lt4j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 21:39:52.341807 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:52.341272 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" podUID="9be23a49-ab91-409d-b131-cb4a917a58e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 21:39:53.260500 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:53.260468 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:53.273210 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:53.273178 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:53.340973 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:53.340929 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lt4j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 21:39:53.341181 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:53.341003 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" podUID="9be23a49-ab91-409d-b131-cb4a917a58e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 21:39:54.275210 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:54.275172 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" Nov 03 21:39:54.340884 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:54.340853 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lt4j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 21:39:54.341061 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:54.340916 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" podUID="9be23a49-ab91-409d-b131-cb4a917a58e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 21:39:55.500431 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:55.500395 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:55.501121 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:55.501094 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:39:55.502098 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:39:55.502074 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" Nov 03 21:40:01.256611 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:01.256567 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:40:10.786794 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:40:10.786743 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:40:18.508044 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:18.508009 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zlxvh"] Nov 03 21:40:18.513546 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:18.513468 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zlxvh" Nov 03 21:40:18.531797 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:18.531765 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zlxvh"] Nov 03 21:40:18.590998 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:18.590964 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 21:40:18.591214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:18.591141 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4pxx4\"" Nov 03 21:40:18.681136 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:18.681098 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63ff686-c19d-485e-b503-22cc297aa852-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zlxvh\" (UID: \"e63ff686-c19d-485e-b503-22cc297aa852\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zlxvh" Nov 03 21:40:18.681313 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:18.681145 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqjsg\" (UniqueName: \"kubernetes.io/projected/e63ff686-c19d-485e-b503-22cc297aa852-kube-api-access-mqjsg\") pod \"payload-pre-processing-7b98bb678c-zlxvh\" (UID: \"e63ff686-c19d-485e-b503-22cc297aa852\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zlxvh" Nov 03 21:40:18.782378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:18.782293 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63ff686-c19d-485e-b503-22cc297aa852-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zlxvh\" (UID: \"e63ff686-c19d-485e-b503-22cc297aa852\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zlxvh" Nov 03 21:40:18.782378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:18.782332 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqjsg\" (UniqueName: \"kubernetes.io/projected/e63ff686-c19d-485e-b503-22cc297aa852-kube-api-access-mqjsg\") pod \"payload-pre-processing-7b98bb678c-zlxvh\" (UID: \"e63ff686-c19d-485e-b503-22cc297aa852\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zlxvh" Nov 03 21:40:18.783097 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:18.783076 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63ff686-c19d-485e-b503-22cc297aa852-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zlxvh\" (UID: \"e63ff686-c19d-485e-b503-22cc297aa852\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zlxvh" Nov 03 21:40:18.803093 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:18.803062 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqjsg\" (UniqueName: \"kubernetes.io/projected/e63ff686-c19d-485e-b503-22cc297aa852-kube-api-access-mqjsg\") pod \"payload-pre-processing-7b98bb678c-zlxvh\" (UID: \"e63ff686-c19d-485e-b503-22cc297aa852\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zlxvh" Nov 03 21:40:18.826687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:18.826635 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zlxvh" Nov 03 21:40:19.122738 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:40:19.122694 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63ff686_c19d_485e_b503_22cc297aa852.slice/crio-011feb35d4ce134d36495637dca8ab0dbb46f37ee281eea45f818795bfd9e30a WatchSource:0}: Error finding container 011feb35d4ce134d36495637dca8ab0dbb46f37ee281eea45f818795bfd9e30a: Status 404 returned error can't find the container with id 011feb35d4ce134d36495637dca8ab0dbb46f37ee281eea45f818795bfd9e30a Nov 03 21:40:19.122738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:19.122711 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zlxvh"] Nov 03 21:40:19.125437 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:19.125416 5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:40:19.475784 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:19.475746 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zlxvh" event={"ID":"e63ff686-c19d-485e-b503-22cc297aa852","Type":"ContainerStarted","Data":"011feb35d4ce134d36495637dca8ab0dbb46f37ee281eea45f818795bfd9e30a"} Nov 03 21:40:21.491890 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:21.491851 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zlxvh" event={"ID":"e63ff686-c19d-485e-b503-22cc297aa852","Type":"ContainerStarted","Data":"cd1f8110b54379b94419ade308e73f968ae84719ebc648a8bbf4f23498a9a609"} Nov 03 21:40:21.492338 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:21.491914 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zlxvh" Nov 03 21:40:21.534345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:21.534274 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zlxvh" podStartSLOduration=1.547070725 podStartE2EDuration="3.534254202s" totalImagesPullingTime="1.987183477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:21.532192793 +0000 UTC m=+637.039421098" watchObservedRunningTime="2026-11-03 21:40:21.534254202 +0000 UTC m=+637.041482483" Nov 03 21:40:32.501143 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:40:32.501112 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zlxvh" Nov 03 21:40:40.777001 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:40:40.776956 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:41:10.776443 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:41:10.776370 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:41:39.215816 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.215778 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr"] Nov 03 21:41:39.220752 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.220723 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.243898 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.243862 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr"] Nov 03 21:41:39.253438 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.253401 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-27fdf4-openshift-default-dockercfg-5cb9j\"" Nov 03 21:41:39.395808 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.395769 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istiod-ca-cert\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.395983 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.395817 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-workload-certs\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.395983 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.395850 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-data\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.396081 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.395969 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-token\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.396145 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.396082 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-workload-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.396202 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.396173 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-podinfo\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.396258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.396245 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-credential-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.396326 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.396311 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-envoy\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.396380 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.396340 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkpdc\" (UniqueName: \"kubernetes.io/projected/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-kube-api-access-xkpdc\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.497919 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.497832 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-workload-certs\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.497919 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.497881 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-data\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.497919 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.497904 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-token\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.498189 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.497938 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-workload-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.498189 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.497982 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-podinfo\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.498189 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.498020 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-credential-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.498189 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.498063 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-envoy\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.498189 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.498089 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkpdc\" (UniqueName: \"kubernetes.io/projected/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-kube-api-access-xkpdc\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.498449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.498210 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istiod-ca-cert\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.498449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.498299 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-data\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.498449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.498349 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-workload-certs\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.498602 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.498586 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-credential-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.498913 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.498890 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-workload-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.498989 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.498909 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istiod-ca-cert\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.501165 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.501130 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-envoy\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.502419 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.502386 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-podinfo\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.518997 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.518964 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-token\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.519295 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.519270 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkpdc\" (UniqueName: \"kubernetes.io/projected/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-kube-api-access-xkpdc\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.537241 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.537214 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:39.596498 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.596456 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k"] Nov 03 21:41:39.632116 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.631226 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.632410 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.631126 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k"] Nov 03 21:41:39.647694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.647648 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-05bc27-openshift-default-dockercfg-bxn7k\"" Nov 03 21:41:39.809202 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.809102 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-credential-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.809202 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.809164 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-istio-envoy\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.809387 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.809263 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-istio-data\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.809387 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.809315 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfps4\" (UniqueName: \"kubernetes.io/projected/523ee582-4304-412b-8269-cf9cf716c8a2-kube-api-access-qfps4\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.809474 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.809404 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-workload-certs\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.809474 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.809453 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-workload-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.809547 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.809522 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523ee582-4304-412b-8269-cf9cf716c8a2-istiod-ca-cert\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.809547 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.809544 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523ee582-4304-412b-8269-cf9cf716c8a2-istio-token\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.809748 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.809729 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523ee582-4304-412b-8269-cf9cf716c8a2-istio-podinfo\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.906301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.906263 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr"] Nov 03 21:41:39.907916 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:41:39.907882 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3eac22_c52b_48e7_bc7e_eddebf306cb5.slice/crio-4a6c0b110599ec08ace65cd9a53a59cea78cdd05fad9050fceac221973c307e3 WatchSource:0}: Error finding container 4a6c0b110599ec08ace65cd9a53a59cea78cdd05fad9050fceac221973c307e3: Status 404 returned error can't find the container with id 4a6c0b110599ec08ace65cd9a53a59cea78cdd05fad9050fceac221973c307e3 Nov 03 21:41:39.910358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.910332 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523ee582-4304-412b-8269-cf9cf716c8a2-istiod-ca-cert\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.910476 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.910369 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523ee582-4304-412b-8269-cf9cf716c8a2-istio-token\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.910476 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.910410 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523ee582-4304-412b-8269-cf9cf716c8a2-istio-podinfo\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.910476 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.910431 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:41:39.910476 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.910462 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-credential-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.910714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.910507 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-istio-envoy\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.910714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.910520 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:41:39.910714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.910613 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:41:39.910714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.910678 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-istio-data\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.910924 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.910732 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfps4\" (UniqueName: \"kubernetes.io/projected/523ee582-4304-412b-8269-cf9cf716c8a2-kube-api-access-qfps4\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.910924 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.910781 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-workload-certs\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.910924 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.910820 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-workload-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.911765 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.911082 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523ee582-4304-412b-8269-cf9cf716c8a2-istiod-ca-cert\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.911765 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.911390 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-workload-certs\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.911765 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.911440 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-credential-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.911765 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.911714 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-workload-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.912065 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.911934 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-istio-data\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.917701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.917350 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-istio-envoy\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.917701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.917566 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523ee582-4304-412b-8269-cf9cf716c8a2-istio-podinfo\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.960770 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.960704 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfps4\" (UniqueName: \"kubernetes.io/projected/523ee582-4304-412b-8269-cf9cf716c8a2-kube-api-access-qfps4\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.966306 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.966240 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523ee582-4304-412b-8269-cf9cf716c8a2-istio-token\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:39.970678 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:39.970623 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h"] Nov 03 21:41:40.006323 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.005915 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h"] Nov 03 21:41:40.006323 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.005969 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" event={"ID":"4b3eac22-c52b-48e7-bc7e-eddebf306cb5","Type":"ContainerStarted","Data":"4a6c0b110599ec08ace65cd9a53a59cea78cdd05fad9050fceac221973c307e3"} Nov 03 21:41:40.006323 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.006089 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.023812 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.023597 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-dd5620-openshift-default-dockercfg-cm274\"" Nov 03 21:41:40.114924 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.114883 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istiod-ca-cert\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.114924 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.114937 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-workload-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.115189 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.115059 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-envoy\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.115189 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.115095 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-workload-certs\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.115189 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.115183 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-podinfo\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.115363 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.115231 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-token\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.115363 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.115322 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-credential-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.115477 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.115370 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fq4l\" (UniqueName: \"kubernetes.io/projected/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-kube-api-access-6fq4l\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.115477 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.115435 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-data\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.217040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.216997 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fq4l\" (UniqueName: \"kubernetes.io/projected/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-kube-api-access-6fq4l\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.217543 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.217075 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-data\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.217543 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.217210 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istiod-ca-cert\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.217543 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.217276 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-workload-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.217543 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.217311 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-envoy\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.217543 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.217351 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-workload-certs\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.217543 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.217414 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-podinfo\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.217543 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.217466 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-data\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.217543 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.217470 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-token\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.217543 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.217534 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-credential-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.218061 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.217614 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-workload-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.218061 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.217928 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-credential-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.218397 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.218370 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istiod-ca-cert\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.218533 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.218410 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-workload-certs\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.220059 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.220033 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-envoy\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.220211 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.220193 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-podinfo\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.238349 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.238316 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fq4l\" (UniqueName: \"kubernetes.io/projected/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-kube-api-access-6fq4l\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.238654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.238625 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-token\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.251443 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.251412 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:40.336269 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.335983 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:40.567321 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.567288 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k"] Nov 03 21:41:40.568238 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:41:40.568202 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523ee582_4304_412b_8269_cf9cf716c8a2.slice/crio-428e6e3e9c745cce28f5861cc6070c37c0e819c4b1a4f410eb17a2a3a7cd5f28 WatchSource:0}: Error finding container 428e6e3e9c745cce28f5861cc6070c37c0e819c4b1a4f410eb17a2a3a7cd5f28: Status 404 returned error can't find the container with id 428e6e3e9c745cce28f5861cc6070c37c0e819c4b1a4f410eb17a2a3a7cd5f28 Nov 03 21:41:40.570935 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.570897 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:41:40.571030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.570987 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:41:40.571083 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.571070 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:41:40.643990 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.643960 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h"] Nov 03 21:41:40.644849 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:41:40.644825 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dcc2293_9b7c_4381_92f2_1657ee9a1e29.slice/crio-cea920656c09828b68820e8229412800bb1aa4b633675e0fc4b50ee072b22039 WatchSource:0}: Error finding container cea920656c09828b68820e8229412800bb1aa4b633675e0fc4b50ee072b22039: Status 404 returned error can't find the container with id cea920656c09828b68820e8229412800bb1aa4b633675e0fc4b50ee072b22039 Nov 03 21:41:40.647401 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.647371 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:41:40.647496 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.647433 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:41:40.647496 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:40.647482 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:41:40.791917 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:41:40.791842 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:41:41.005920 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:41.005878 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" event={"ID":"4dcc2293-9b7c-4381-92f2-1657ee9a1e29","Type":"ContainerStarted","Data":"24a652966f936844adcfac8963d00e9fb6751cc2993ee5a29dd0344e8236fa58"} Nov 03 21:41:41.006138 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:41.005945 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" event={"ID":"4dcc2293-9b7c-4381-92f2-1657ee9a1e29","Type":"ContainerStarted","Data":"cea920656c09828b68820e8229412800bb1aa4b633675e0fc4b50ee072b22039"} Nov 03 21:41:41.007980 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:41.007946 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" event={"ID":"523ee582-4304-412b-8269-cf9cf716c8a2","Type":"ContainerStarted","Data":"c28a0fb769cde3626a45a18072804d306f6007c6ed852a07ba252a3e66a99299"} Nov 03 21:41:41.008145 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:41.008001 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" event={"ID":"523ee582-4304-412b-8269-cf9cf716c8a2","Type":"ContainerStarted","Data":"428e6e3e9c745cce28f5861cc6070c37c0e819c4b1a4f410eb17a2a3a7cd5f28"} Nov 03 21:41:41.009699 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:41.009673 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" event={"ID":"4b3eac22-c52b-48e7-bc7e-eddebf306cb5","Type":"ContainerStarted","Data":"8b7b7e6859be7f4cf7c3c39d92f79693b071ca823f9dc7ea42d91ebc41a30b1e"} Nov 03 21:41:41.053290 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:41.053230 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" podStartSLOduration=2.05321619 podStartE2EDuration="2.05321619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:41.051484269 +0000 UTC m=+716.558712550" watchObservedRunningTime="2026-11-03 21:41:41.05321619 +0000 UTC m=+716.560444467" Nov 03 21:41:41.097896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:41.097843 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" podStartSLOduration=2.097829047 podStartE2EDuration="2.097829047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:41.095225738 +0000 UTC m=+716.602454019" watchObservedRunningTime="2026-11-03 21:41:41.097829047 +0000 UTC m=+716.605057327" Nov 03 21:41:41.140790 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:41.140739 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" podStartSLOduration=2.140723432 podStartE2EDuration="2.140723432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:41.137935412 +0000 UTC m=+716.645163707" watchObservedRunningTime="2026-11-03 21:41:41.140723432 +0000 UTC m=+716.647951712" Nov 03 21:41:41.251912 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:41.251878 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:41.257038 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:41.256968 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:41.337197 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:41.337161 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:41.343105 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:41.343076 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:41.537708 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:41.537588 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:41.543013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:41.542986 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:42.017387 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:42.017358 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:42.017593 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:42.017411 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:42.017593 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:42.017436 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:42.018433 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:42.018358 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:41:42.018433 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:42.018422 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:41:42.018604 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:42.018578 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:41:44.453502 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:44.453459 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh"] Nov 03 21:41:44.458983 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:44.458952 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" Nov 03 21:41:44.475448 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:44.475419 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stg48\" (UniqueName: \"kubernetes.io/projected/cabaa235-b28f-41a4-ad06-a10e60adc05e-kube-api-access-stg48\") pod \"payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh\" (UID: \"cabaa235-b28f-41a4-ad06-a10e60adc05e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" Nov 03 21:41:44.475612 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:44.475554 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabaa235-b28f-41a4-ad06-a10e60adc05e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh\" (UID: \"cabaa235-b28f-41a4-ad06-a10e60adc05e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" Nov 03 21:41:44.483424 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:44.483391 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh"] Nov 03 21:41:44.493698 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:44.493643 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-4eced5\"" Nov 03 21:41:44.545874 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:44.545838 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-4eced5-dockercfg-t2k4b\"" Nov 03 21:41:44.576484 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:44.576443 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stg48\" (UniqueName: \"kubernetes.io/projected/cabaa235-b28f-41a4-ad06-a10e60adc05e-kube-api-access-stg48\") pod \"payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh\" (UID: \"cabaa235-b28f-41a4-ad06-a10e60adc05e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" Nov 03 21:41:44.576738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:44.576611 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabaa235-b28f-41a4-ad06-a10e60adc05e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh\" (UID: \"cabaa235-b28f-41a4-ad06-a10e60adc05e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" Nov 03 21:41:44.591608 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:44.591577 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-4eced5\"" Nov 03 21:41:44.598535 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:44.598503 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabaa235-b28f-41a4-ad06-a10e60adc05e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh\" (UID: \"cabaa235-b28f-41a4-ad06-a10e60adc05e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" Nov 03 21:41:44.602475 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:44.602432 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stg48\" (UniqueName: \"kubernetes.io/projected/cabaa235-b28f-41a4-ad06-a10e60adc05e-kube-api-access-stg48\") pod \"payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh\" (UID: \"cabaa235-b28f-41a4-ad06-a10e60adc05e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" Nov 03 21:41:44.789791 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:44.789707 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-4eced5-dockercfg-t2k4b\"" Nov 03 21:41:44.794754 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:44.794737 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" Nov 03 21:41:45.151849 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:45.151735 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh"] Nov 03 21:41:45.153762 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:41:45.153727 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabaa235_b28f_41a4_ad06_a10e60adc05e.slice/crio-710ec52de254eb27408f9c1ddffb671ebd83f75c0b6f6b9f30003458508b3f4e WatchSource:0}: Error finding container 710ec52de254eb27408f9c1ddffb671ebd83f75c0b6f6b9f30003458508b3f4e: Status 404 returned error can't find the container with id 710ec52de254eb27408f9c1ddffb671ebd83f75c0b6f6b9f30003458508b3f4e Nov 03 21:41:46.051271 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:46.051226 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" event={"ID":"cabaa235-b28f-41a4-ad06-a10e60adc05e","Type":"ContainerStarted","Data":"8702e5d74d46857bffc7c087b017643103bdd1e81d44ab751757d55c43f01d41"} Nov 03 21:41:46.051779 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:46.051304 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" event={"ID":"cabaa235-b28f-41a4-ad06-a10e60adc05e","Type":"ContainerStarted","Data":"710ec52de254eb27408f9c1ddffb671ebd83f75c0b6f6b9f30003458508b3f4e"} Nov 03 21:41:46.051779 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:46.051338 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" Nov 03 21:41:46.096627 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:46.096551 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" podStartSLOduration=2.096278589 podStartE2EDuration="2.096278589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:46.093278379 +0000 UTC m=+721.600506664" watchObservedRunningTime="2026-11-03 21:41:46.096278589 +0000 UTC m=+721.603506878" Nov 03 21:41:48.283384 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:48.283342 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8"] Nov 03 21:41:48.288495 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:48.288465 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" Nov 03 21:41:48.308256 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:48.308222 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8"] Nov 03 21:41:48.336152 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:48.336120 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05ce451-3614-4af6-88e8-8793cf631ea1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8\" (UID: \"f05ce451-3614-4af6-88e8-8793cf631ea1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" Nov 03 21:41:48.336384 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:48.336363 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r52lg\" (UniqueName: \"kubernetes.io/projected/f05ce451-3614-4af6-88e8-8793cf631ea1-kube-api-access-r52lg\") pod \"payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8\" (UID: \"f05ce451-3614-4af6-88e8-8793cf631ea1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" Nov 03 21:41:48.370486 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:48.370455 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-0a6da1-dockercfg-5744c\"" Nov 03 21:41:48.374115 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:48.374086 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-0a6da1\"" Nov 03 21:41:48.437809 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:48.437771 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05ce451-3614-4af6-88e8-8793cf631ea1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8\" (UID: \"f05ce451-3614-4af6-88e8-8793cf631ea1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" Nov 03 21:41:48.437809 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:48.437813 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r52lg\" (UniqueName: \"kubernetes.io/projected/f05ce451-3614-4af6-88e8-8793cf631ea1-kube-api-access-r52lg\") pod \"payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8\" (UID: \"f05ce451-3614-4af6-88e8-8793cf631ea1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" Nov 03 21:41:48.438845 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:48.438822 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05ce451-3614-4af6-88e8-8793cf631ea1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8\" (UID: \"f05ce451-3614-4af6-88e8-8793cf631ea1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" Nov 03 21:41:48.458150 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:48.458120 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r52lg\" (UniqueName: \"kubernetes.io/projected/f05ce451-3614-4af6-88e8-8793cf631ea1-kube-api-access-r52lg\") pod \"payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8\" (UID: \"f05ce451-3614-4af6-88e8-8793cf631ea1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" Nov 03 21:41:48.618322 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:48.618231 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" Nov 03 21:41:48.930558 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:48.930446 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8"] Nov 03 21:41:48.964813 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:41:48.964778 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05ce451_3614_4af6_88e8_8793cf631ea1.slice/crio-93ac4ca2b0d4f7222e2549a12ef64cde3a837c29d9c0f7addcd3b7c0a3a12983 WatchSource:0}: Error finding container 93ac4ca2b0d4f7222e2549a12ef64cde3a837c29d9c0f7addcd3b7c0a3a12983: Status 404 returned error can't find the container with id 93ac4ca2b0d4f7222e2549a12ef64cde3a837c29d9c0f7addcd3b7c0a3a12983 Nov 03 21:41:49.074921 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.074879 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" event={"ID":"f05ce451-3614-4af6-88e8-8793cf631ea1","Type":"ContainerStarted","Data":"777cab1a0a4603d3b57dbc3d194b833a766e06e25e54dcefefd054ae152d1276"} Nov 03 21:41:49.075108 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.074949 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" event={"ID":"f05ce451-3614-4af6-88e8-8793cf631ea1","Type":"ContainerStarted","Data":"93ac4ca2b0d4f7222e2549a12ef64cde3a837c29d9c0f7addcd3b7c0a3a12983"} Nov 03 21:41:49.075108 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.074983 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" Nov 03 21:41:49.086206 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.086173 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr"] Nov 03 21:41:49.090545 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.090520 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" Nov 03 21:41:49.106844 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.106820 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-dd5620-dockercfg-28p9r\"" Nov 03 21:41:49.110546 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.110518 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr"] Nov 03 21:41:49.119151 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.119095 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" podStartSLOduration=1.119080314 podStartE2EDuration="1.119080314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:49.117901082 +0000 UTC m=+724.625129362" watchObservedRunningTime="2026-11-03 21:41:49.119080314 +0000 UTC m=+724.626308597" Nov 03 21:41:49.148534 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.148503 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbnvt\" (UniqueName: \"kubernetes.io/projected/7526f386-4f83-49e7-a3e0-80bb549fec6c-kube-api-access-bbnvt\") pod \"payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr\" (UID: \"7526f386-4f83-49e7-a3e0-80bb549fec6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" Nov 03 21:41:49.148735 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.148680 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7526f386-4f83-49e7-a3e0-80bb549fec6c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr\" (UID: \"7526f386-4f83-49e7-a3e0-80bb549fec6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" Nov 03 21:41:49.167291 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.167262 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-dd5620\"" Nov 03 21:41:49.185600 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.185506 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq"] Nov 03 21:41:49.191239 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.191212 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.229861 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.229827 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-517149-openshift-default-dockercfg-hr4lp\"" Nov 03 21:41:49.233773 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.233734 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq"] Nov 03 21:41:49.249433 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.249396 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-podinfo\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.249720 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.249694 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7526f386-4f83-49e7-a3e0-80bb549fec6c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr\" (UID: \"7526f386-4f83-49e7-a3e0-80bb549fec6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" Nov 03 21:41:49.249828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.249731 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-data\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.249828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.249763 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-workload-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.249958 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.249893 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-envoy\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.250009 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.249959 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcg6n\" (UniqueName: \"kubernetes.io/projected/9507b5d4-33e4-470f-9385-59164cb2e72a-kube-api-access-lcg6n\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.250009 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.249998 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9507b5d4-33e4-470f-9385-59164cb2e72a-istiod-ca-cert\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.250110 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.250033 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-workload-certs\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.250110 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.250063 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-credential-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.250110 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.250087 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-token\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.250247 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.250121 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbnvt\" (UniqueName: \"kubernetes.io/projected/7526f386-4f83-49e7-a3e0-80bb549fec6c-kube-api-access-bbnvt\") pod \"payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr\" (UID: \"7526f386-4f83-49e7-a3e0-80bb549fec6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" Nov 03 21:41:49.251424 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.251401 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7526f386-4f83-49e7-a3e0-80bb549fec6c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr\" (UID: \"7526f386-4f83-49e7-a3e0-80bb549fec6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" Nov 03 21:41:49.273125 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.273098 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbnvt\" (UniqueName: \"kubernetes.io/projected/7526f386-4f83-49e7-a3e0-80bb549fec6c-kube-api-access-bbnvt\") pod \"payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr\" (UID: \"7526f386-4f83-49e7-a3e0-80bb549fec6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" Nov 03 21:41:49.354287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.351213 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-workload-certs\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.354287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.351269 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-credential-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.354287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.351293 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-token\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.354287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.351374 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-podinfo\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.354287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.351585 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-data\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.354287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.351623 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-workload-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.354287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.351688 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-envoy\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.354287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.351748 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcg6n\" (UniqueName: \"kubernetes.io/projected/9507b5d4-33e4-470f-9385-59164cb2e72a-kube-api-access-lcg6n\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.354287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.351787 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9507b5d4-33e4-470f-9385-59164cb2e72a-istiod-ca-cert\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.354287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.353702 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-credential-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.354287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.353807 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9507b5d4-33e4-470f-9385-59164cb2e72a-istiod-ca-cert\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.354287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.354076 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-workload-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.354287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.354103 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-workload-certs\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.355406 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.354407 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-data\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.359248 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.359210 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-envoy\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.359760 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.359536 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-podinfo\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.376076 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.376043 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-token\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.377413 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.377384 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcg6n\" (UniqueName: \"kubernetes.io/projected/9507b5d4-33e4-470f-9385-59164cb2e72a-kube-api-access-lcg6n\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.408889 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.408850 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" Nov 03 21:41:49.507028 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.506994 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:49.728607 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:41:49.728527 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7526f386_4f83_49e7_a3e0_80bb549fec6c.slice/crio-126e1520864d8319a6efabd719ae32a2967c9d5da2ce68e06cb05cf16a98b14a WatchSource:0}: Error finding container 126e1520864d8319a6efabd719ae32a2967c9d5da2ce68e06cb05cf16a98b14a: Status 404 returned error can't find the container with id 126e1520864d8319a6efabd719ae32a2967c9d5da2ce68e06cb05cf16a98b14a Nov 03 21:41:49.728935 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.728907 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr"] Nov 03 21:41:49.839004 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.838978 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq"] Nov 03 21:41:49.840436 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:41:49.840407 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9507b5d4_33e4_470f_9385_59164cb2e72a.slice/crio-2a36549d77b928b62c7579a3bf23f9107b9fce1ff9c0712f5e968a12b5569d35 WatchSource:0}: Error finding container 2a36549d77b928b62c7579a3bf23f9107b9fce1ff9c0712f5e968a12b5569d35: Status 404 returned error can't find the container with id 2a36549d77b928b62c7579a3bf23f9107b9fce1ff9c0712f5e968a12b5569d35 Nov 03 21:41:49.843069 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.843037 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:41:49.843192 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.843112 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:41:49.843192 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:49.843155 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:41:50.082959 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:50.082858 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" event={"ID":"9507b5d4-33e4-470f-9385-59164cb2e72a","Type":"ContainerStarted","Data":"e6787f9d10bc60d58558aa3e2139d296b0b058d4dc1abe95cba826be5e3226f9"} Nov 03 21:41:50.082959 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:50.082923 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" event={"ID":"9507b5d4-33e4-470f-9385-59164cb2e72a","Type":"ContainerStarted","Data":"2a36549d77b928b62c7579a3bf23f9107b9fce1ff9c0712f5e968a12b5569d35"} Nov 03 21:41:50.084877 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:50.084850 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" event={"ID":"7526f386-4f83-49e7-a3e0-80bb549fec6c","Type":"ContainerStarted","Data":"07566408ab0372e0d8235a22dd8ac83e1755ea66c7640050bf921d7d37ff93c8"} Nov 03 21:41:50.085038 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:50.084892 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" event={"ID":"7526f386-4f83-49e7-a3e0-80bb549fec6c","Type":"ContainerStarted","Data":"126e1520864d8319a6efabd719ae32a2967c9d5da2ce68e06cb05cf16a98b14a"} Nov 03 21:41:50.085038 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:50.085021 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" Nov 03 21:41:50.128164 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:50.128084 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" podStartSLOduration=1.128063639 podStartE2EDuration="1.128063639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:50.125780077 +0000 UTC m=+725.633008371" watchObservedRunningTime="2026-11-03 21:41:50.128063639 +0000 UTC m=+725.635291920" Nov 03 21:41:50.170451 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:50.170394 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" podStartSLOduration=1.170380359 podStartE2EDuration="1.170380359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:50.167767996 +0000 UTC m=+725.674996276" watchObservedRunningTime="2026-11-03 21:41:50.170380359 +0000 UTC m=+725.677608659" Nov 03 21:41:50.510705 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:50.509775 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:50.539131 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:50.539093 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:51.091174 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:51.091138 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:51.092148 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:51.092119 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:41:54.693996 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:54.693956 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272"] Nov 03 21:41:54.699060 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:54.699038 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" Nov 03 21:41:54.728172 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:54.728141 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-517149\"" Nov 03 21:41:54.734465 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:54.734431 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272"] Nov 03 21:41:54.739843 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:54.739815 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-517149-dockercfg-g2n5w\"" Nov 03 21:41:54.868798 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:54.846400 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20b45af0-efad-4f42-a998-784bf9f39be9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272\" (UID: \"20b45af0-efad-4f42-a998-784bf9f39be9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" Nov 03 21:41:54.869178 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:54.869150 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bmkd\" (UniqueName: \"kubernetes.io/projected/20b45af0-efad-4f42-a998-784bf9f39be9-kube-api-access-4bmkd\") pod \"payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272\" (UID: \"20b45af0-efad-4f42-a998-784bf9f39be9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" Nov 03 21:41:54.974033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:54.973995 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20b45af0-efad-4f42-a998-784bf9f39be9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272\" (UID: \"20b45af0-efad-4f42-a998-784bf9f39be9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" Nov 03 21:41:54.974228 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:54.974155 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bmkd\" (UniqueName: \"kubernetes.io/projected/20b45af0-efad-4f42-a998-784bf9f39be9-kube-api-access-4bmkd\") pod \"payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272\" (UID: \"20b45af0-efad-4f42-a998-784bf9f39be9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" Nov 03 21:41:54.974723 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:54.974703 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20b45af0-efad-4f42-a998-784bf9f39be9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272\" (UID: \"20b45af0-efad-4f42-a998-784bf9f39be9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" Nov 03 21:41:54.996808 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:54.996773 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bmkd\" (UniqueName: \"kubernetes.io/projected/20b45af0-efad-4f42-a998-784bf9f39be9-kube-api-access-4bmkd\") pod \"payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272\" (UID: \"20b45af0-efad-4f42-a998-784bf9f39be9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" Nov 03 21:41:55.065782 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:55.065746 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" Nov 03 21:41:55.392479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:55.392450 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272"] Nov 03 21:41:55.393791 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:41:55.393760 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b45af0_efad_4f42_a998_784bf9f39be9.slice/crio-d6138f637254794aa8dc7a99586171220106bf4db1ddcca161e79303ae12e766 WatchSource:0}: Error finding container d6138f637254794aa8dc7a99586171220106bf4db1ddcca161e79303ae12e766: Status 404 returned error can't find the container with id d6138f637254794aa8dc7a99586171220106bf4db1ddcca161e79303ae12e766 Nov 03 21:41:56.149766 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:56.149726 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" event={"ID":"20b45af0-efad-4f42-a998-784bf9f39be9","Type":"ContainerStarted","Data":"11d77de13d464a4bbda88b9ec3a2b66b34115d9c427383f1f05aca24c58841fb"} Nov 03 21:41:56.150232 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:56.149790 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" event={"ID":"20b45af0-efad-4f42-a998-784bf9f39be9","Type":"ContainerStarted","Data":"d6138f637254794aa8dc7a99586171220106bf4db1ddcca161e79303ae12e766"} Nov 03 21:41:56.150232 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:56.149976 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" Nov 03 21:41:56.190113 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:56.190056 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" podStartSLOduration=2.190043184 podStartE2EDuration="2.190043184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:56.189793897 +0000 UTC m=+731.697022179" watchObservedRunningTime="2026-11-03 21:41:56.190043184 +0000 UTC m=+731.697271464" Nov 03 21:41:57.062365 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:57.062337 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" Nov 03 21:41:57.696838 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:57.696801 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9"] Nov 03 21:41:57.701422 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:57.701404 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" Nov 03 21:41:57.741063 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:57.741030 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9"] Nov 03 21:41:57.749912 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:57.749887 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e246f9\"" Nov 03 21:41:57.750230 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:57.750213 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e246f9-dockercfg-znz9n\"" Nov 03 21:41:57.819699 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:57.812330 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47dc1e9a-eadd-46d0-8d2f-105d8fd8baba-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9\" (UID: \"47dc1e9a-eadd-46d0-8d2f-105d8fd8baba\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" Nov 03 21:41:57.819699 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:57.812555 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbsjd\" (UniqueName: \"kubernetes.io/projected/47dc1e9a-eadd-46d0-8d2f-105d8fd8baba-kube-api-access-xbsjd\") pod \"payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9\" (UID: \"47dc1e9a-eadd-46d0-8d2f-105d8fd8baba\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" Nov 03 21:41:57.932702 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:57.924396 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbsjd\" (UniqueName: \"kubernetes.io/projected/47dc1e9a-eadd-46d0-8d2f-105d8fd8baba-kube-api-access-xbsjd\") pod \"payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9\" (UID: \"47dc1e9a-eadd-46d0-8d2f-105d8fd8baba\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" Nov 03 21:41:57.932702 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:57.924480 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47dc1e9a-eadd-46d0-8d2f-105d8fd8baba-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9\" (UID: \"47dc1e9a-eadd-46d0-8d2f-105d8fd8baba\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" Nov 03 21:41:57.932702 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:57.925602 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47dc1e9a-eadd-46d0-8d2f-105d8fd8baba-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9\" (UID: \"47dc1e9a-eadd-46d0-8d2f-105d8fd8baba\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" Nov 03 21:41:57.956683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:57.953840 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbsjd\" (UniqueName: \"kubernetes.io/projected/47dc1e9a-eadd-46d0-8d2f-105d8fd8baba-kube-api-access-xbsjd\") pod \"payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9\" (UID: \"47dc1e9a-eadd-46d0-8d2f-105d8fd8baba\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" Nov 03 21:41:58.045938 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:58.045896 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" Nov 03 21:41:58.377742 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:58.377712 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9"] Nov 03 21:41:58.379141 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:41:58.379116 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47dc1e9a_eadd_46d0_8d2f_105d8fd8baba.slice/crio-144971c6a0d0d37c02eab9b2a36efc28846e849a7020b9c5200a2cf387ebffb3 WatchSource:0}: Error finding container 144971c6a0d0d37c02eab9b2a36efc28846e849a7020b9c5200a2cf387ebffb3: Status 404 returned error can't find the container with id 144971c6a0d0d37c02eab9b2a36efc28846e849a7020b9c5200a2cf387ebffb3 Nov 03 21:41:59.176989 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:59.176950 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" event={"ID":"47dc1e9a-eadd-46d0-8d2f-105d8fd8baba","Type":"ContainerStarted","Data":"b9c0d27439af14c0779fcb4ad70bdb90f3285d4ad546bc7b121284090e01a1fd"} Nov 03 21:41:59.177404 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:59.177000 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" event={"ID":"47dc1e9a-eadd-46d0-8d2f-105d8fd8baba","Type":"ContainerStarted","Data":"144971c6a0d0d37c02eab9b2a36efc28846e849a7020b9c5200a2cf387ebffb3"} Nov 03 21:41:59.177404 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:41:59.177155 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" Nov 03 21:42:00.088710 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:00.088657 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" Nov 03 21:42:00.133765 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:00.133641 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" podStartSLOduration=3.133623483 podStartE2EDuration="3.133623483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:59.226941015 +0000 UTC m=+734.734169312" watchObservedRunningTime="2026-11-03 21:42:00.133623483 +0000 UTC m=+735.640851764" Nov 03 21:42:01.094932 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:01.094899 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" Nov 03 21:42:05.254112 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:05.254075 5041 generic.go:356] "Generic (PLEG): container finished" podID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" containerID="8b7b7e6859be7f4cf7c3c39d92f79693b071ca823f9dc7ea42d91ebc41a30b1e" exitCode=0 Nov 03 21:42:05.254557 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:05.254150 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" event={"ID":"4b3eac22-c52b-48e7-bc7e-eddebf306cb5","Type":"ContainerDied","Data":"8b7b7e6859be7f4cf7c3c39d92f79693b071ca823f9dc7ea42d91ebc41a30b1e"} Nov 03 21:42:05.254776 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:05.254747 5041 scope.go:122] "RemoveContainer" containerID="8b7b7e6859be7f4cf7c3c39d92f79693b071ca823f9dc7ea42d91ebc41a30b1e" Nov 03 21:42:05.256281 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:05.256255 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:42:05.256356 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:05.256336 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:42:05.256592 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:05.256559 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:42:06.378683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:06.374029 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" event={"ID":"4b3eac22-c52b-48e7-bc7e-eddebf306cb5","Type":"ContainerStarted","Data":"6491ab9a2d3123365fcf19697178a8895552eae48eb6fb8dadef33e6f11ffff2"} Nov 03 21:42:06.538097 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:06.538058 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:42:06.538444 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:06.538424 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 21:42:06.538590 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:06.538565 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" podUID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 21:42:07.167803 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:07.166674 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" Nov 03 21:42:07.539688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:07.538056 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 21:42:07.539688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:07.538124 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" podUID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 21:42:08.539513 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:08.539040 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 21:42:08.539513 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:08.539114 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" podUID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 21:42:09.566034 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:09.565999 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 21:42:09.574790 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:09.574742 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" podUID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 21:42:10.186807 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:10.186779 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" Nov 03 21:42:10.549694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:10.547788 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 21:42:10.549694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:10.547859 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" podUID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 21:42:10.798382 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:42:10.798339 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:42:11.538725 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:11.538225 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 21:42:11.538725 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:11.538301 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" podUID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 21:42:12.020791 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:12.019765 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:42:12.550277 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:12.549954 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:42:12.554410 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:12.554383 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:42:13.062887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:13.062827 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:42:13.063380 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:13.062925 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" podUID="523ee582-4304-412b-8269-cf9cf716c8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:42:13.074851 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:13.074808 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:42:13.075020 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:13.074879 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" podUID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:42:14.606837 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:14.606803 5041 generic.go:356] "Generic (PLEG): container finished" podID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" containerID="24a652966f936844adcfac8963d00e9fb6751cc2993ee5a29dd0344e8236fa58" exitCode=0 Nov 03 21:42:14.607398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:14.606843 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" event={"ID":"4dcc2293-9b7c-4381-92f2-1657ee9a1e29","Type":"ContainerDied","Data":"24a652966f936844adcfac8963d00e9fb6751cc2993ee5a29dd0344e8236fa58"} Nov 03 21:42:14.607398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:14.607337 5041 scope.go:122] "RemoveContainer" containerID="24a652966f936844adcfac8963d00e9fb6751cc2993ee5a29dd0344e8236fa58" Nov 03 21:42:14.613690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:14.609460 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:42:14.613690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:14.609559 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:42:14.613690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:14.609611 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:42:15.659686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:15.646195 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" event={"ID":"4dcc2293-9b7c-4381-92f2-1657ee9a1e29","Type":"ContainerStarted","Data":"294a95b54fb0ae20b17cbeaad31f1d59674f2a7b21ba002703a77e6c206136a5"} Nov 03 21:42:16.348559 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:16.347798 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:42:16.348559 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:16.348232 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:42:16.348559 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:16.348284 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" podUID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 21:42:17.337167 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:17.337071 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:42:17.338238 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:17.337806 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" podUID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 21:42:17.696142 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:17.696096 5041 generic.go:356] "Generic (PLEG): container finished" podID="523ee582-4304-412b-8269-cf9cf716c8a2" containerID="c28a0fb769cde3626a45a18072804d306f6007c6ed852a07ba252a3e66a99299" exitCode=0 Nov 03 21:42:17.696336 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:17.696158 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" event={"ID":"523ee582-4304-412b-8269-cf9cf716c8a2","Type":"ContainerDied","Data":"c28a0fb769cde3626a45a18072804d306f6007c6ed852a07ba252a3e66a99299"} Nov 03 21:42:17.696754 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:17.696731 5041 scope.go:122] "RemoveContainer" containerID="c28a0fb769cde3626a45a18072804d306f6007c6ed852a07ba252a3e66a99299" Nov 03 21:42:17.698291 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:17.698264 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:42:17.698385 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:17.698335 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:42:17.698385 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:17.698372 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:42:18.358504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:18.340774 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:42:18.358504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:18.340837 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" podUID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 21:42:18.707021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:18.706902 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" event={"ID":"523ee582-4304-412b-8269-cf9cf716c8a2","Type":"ContainerStarted","Data":"3b579db11de64b67ef6d213911848487c7d69d88904546ee67b0dce15fbc75a4"} Nov 03 21:42:19.267472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:19.267392 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:42:19.268084 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:19.267783 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:42:19.268084 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:19.267836 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" podUID="523ee582-4304-412b-8269-cf9cf716c8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 21:42:19.341688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:19.341426 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:42:19.341688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:19.341502 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" podUID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 21:42:20.256689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:20.256536 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:42:20.256689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:20.256615 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" podUID="523ee582-4304-412b-8269-cf9cf716c8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 21:42:20.342571 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:20.342095 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 21:42:20.342571 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:20.342163 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" podUID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 21:42:21.252892 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:21.252849 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:42:21.253080 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:21.252917 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" podUID="523ee582-4304-412b-8269-cf9cf716c8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 21:42:21.348722 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:21.347739 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:42:21.348722 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:21.348643 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:42:21.350766 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:21.350706 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:42:22.133183 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:22.132840 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:42:22.133183 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:22.132903 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" podUID="9507b5d4-33e4-470f-9385-59164cb2e72a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:42:22.252040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:22.252004 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:42:22.252225 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:22.252063 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" podUID="523ee582-4304-412b-8269-cf9cf716c8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 21:42:23.252208 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:23.252168 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:42:23.252789 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:23.252222 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" podUID="523ee582-4304-412b-8269-cf9cf716c8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 21:42:23.797138 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:23.797099 5041 generic.go:356] "Generic (PLEG): container finished" podID="9507b5d4-33e4-470f-9385-59164cb2e72a" containerID="e6787f9d10bc60d58558aa3e2139d296b0b058d4dc1abe95cba826be5e3226f9" exitCode=0 Nov 03 21:42:23.797365 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:23.797174 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" event={"ID":"9507b5d4-33e4-470f-9385-59164cb2e72a","Type":"ContainerDied","Data":"e6787f9d10bc60d58558aa3e2139d296b0b058d4dc1abe95cba826be5e3226f9"} Nov 03 21:42:23.797746 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:23.797650 5041 scope.go:122] "RemoveContainer" containerID="e6787f9d10bc60d58558aa3e2139d296b0b058d4dc1abe95cba826be5e3226f9" Nov 03 21:42:23.799446 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:23.799388 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:42:23.799541 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:23.799497 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:42:23.799541 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:23.799534 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:42:24.252430 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:24.252394 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 21:42:24.253116 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:24.253064 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" podUID="523ee582-4304-412b-8269-cf9cf716c8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 21:42:24.805265 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:24.805229 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" event={"ID":"9507b5d4-33e4-470f-9385-59164cb2e72a","Type":"ContainerStarted","Data":"d4a7661eb514e5931ceb633572b4ab35f215cd778792a6cc46fd057c387b92ce"} Nov 03 21:42:25.258602 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:25.258156 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:42:25.266694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:25.265018 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:42:25.266694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:25.266489 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:42:25.507560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:25.507121 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:42:25.507560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:25.507268 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 21:42:25.507560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:25.507325 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" podUID="9507b5d4-33e4-470f-9385-59164cb2e72a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 21:42:26.507740 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:26.507637 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 21:42:26.508238 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:26.507730 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" podUID="9507b5d4-33e4-470f-9385-59164cb2e72a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 21:42:27.507371 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:27.507335 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 21:42:27.507545 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:27.507394 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" podUID="9507b5d4-33e4-470f-9385-59164cb2e72a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 21:42:28.517687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.512060 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 21:42:28.517687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.512131 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" podUID="9507b5d4-33e4-470f-9385-59164cb2e72a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 21:42:28.604326 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.604289 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q"] Nov 03 21:42:28.640891 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.640862 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.656555 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.656524 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q"] Nov 03 21:42:28.672528 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.672501 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd5620\"/\"default-dockercfg-mwz2f\"" Nov 03 21:42:28.672528 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.672501 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd5620\"/\"simulator-w3-dd5620-kserve-self-signed-certs\"" Nov 03 21:42:28.672528 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.672527 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-dd5620\"/\"openshift-service-ca.crt\"" Nov 03 21:42:28.672831 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.672552 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-dd5620\"/\"kube-root-ca.crt\"" Nov 03 21:42:28.696255 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.696226 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-home\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.696255 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.696262 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-dshm\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.696475 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.696350 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-tmp-dir\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.696475 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.696411 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-tls-certs\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.696475 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.696437 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-model-cache\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.696649 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.696537 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd46h\" (UniqueName: \"kubernetes.io/projected/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-kube-api-access-hd46h\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.798334 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.798239 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-home\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.798334 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.798297 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-dshm\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.798587 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.798362 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-tmp-dir\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.798587 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.798431 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-tls-certs\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.798733 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.798568 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-model-cache\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.798896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.798870 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd46h\" (UniqueName: \"kubernetes.io/projected/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-kube-api-access-hd46h\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.799449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.799419 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-model-cache\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.799567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.799462 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-home\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.799567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.799483 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-tmp-dir\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.801944 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.801894 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-dshm\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.802325 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.802301 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-tls-certs\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.845693 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.845647 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd46h\" (UniqueName: \"kubernetes.io/projected/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-kube-api-access-hd46h\") pod \"simulator-w3-dd5620-kserve-55969759c-ftz8q\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:28.957392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:28.957352 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:29.284252 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:29.284222 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q"] Nov 03 21:42:29.286313 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:42:29.286281 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fd6761c_c37e_4d69_83a6_7ea3ee5e9118.slice/crio-8c5705b93b696818de2b1c016e53799e6733a0f830055513c4a98ae0004f2d1e WatchSource:0}: Error finding container 8c5705b93b696818de2b1c016e53799e6733a0f830055513c4a98ae0004f2d1e: Status 404 returned error can't find the container with id 8c5705b93b696818de2b1c016e53799e6733a0f830055513c4a98ae0004f2d1e Nov 03 21:42:29.508137 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:29.507642 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 21:42:29.508137 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:29.507747 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" podUID="9507b5d4-33e4-470f-9385-59164cb2e72a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 21:42:29.895232 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:29.895197 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" event={"ID":"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118","Type":"ContainerStarted","Data":"22ae285ab38147b029a3cbe3b9ca94ee4537f4a36a7170afa4239c55038e77e9"} Nov 03 21:42:29.895679 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:29.895247 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" event={"ID":"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118","Type":"ContainerStarted","Data":"8c5705b93b696818de2b1c016e53799e6733a0f830055513c4a98ae0004f2d1e"} Nov 03 21:42:29.947417 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:29.946227 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podStartSLOduration=1.946203143 podStartE2EDuration="1.946203143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:29.941832053 +0000 UTC m=+765.449060413" watchObservedRunningTime="2026-11-03 21:42:29.946203143 +0000 UTC m=+765.453431428" Nov 03 21:42:29.957671 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:29.957629 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:30.512484 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:30.512451 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:42:30.512851 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:30.512823 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:42:30.513591 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:30.513572 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:42:36.354031 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.353993 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5"] Nov 03 21:42:36.358220 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.358190 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.375442 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.375358 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5"] Nov 03 21:42:36.384843 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.384798 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-517149\"/\"simulator-w6-517149-kserve-self-signed-certs\"" Nov 03 21:42:36.385030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.384871 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-517149\"/\"default-dockercfg-jbw89\"" Nov 03 21:42:36.389852 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.389818 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-517149\"/\"openshift-service-ca.crt\"" Nov 03 21:42:36.390018 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.389878 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-517149\"/\"kube-root-ca.crt\"" Nov 03 21:42:36.501782 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.501745 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-model-cache\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.501976 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.501799 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9cm\" (UniqueName: \"kubernetes.io/projected/176dd3a2-8222-4882-97d7-27e51c5cd526-kube-api-access-4t9cm\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.501976 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.501904 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-home\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.501976 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.501953 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-tmp-dir\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.502091 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.502019 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-dshm\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.502091 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.502059 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176dd3a2-8222-4882-97d7-27e51c5cd526-tls-certs\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.602925 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.602881 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176dd3a2-8222-4882-97d7-27e51c5cd526-tls-certs\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.603161 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.603028 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-model-cache\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.603161 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.603074 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9cm\" (UniqueName: \"kubernetes.io/projected/176dd3a2-8222-4882-97d7-27e51c5cd526-kube-api-access-4t9cm\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.603161 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.603118 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-home\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.603161 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.603162 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-tmp-dir\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.603409 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.603202 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-dshm\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.604606 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.604525 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-tmp-dir\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.604830 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.604808 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-model-cache\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.605074 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.605055 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-home\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.607429 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.607400 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176dd3a2-8222-4882-97d7-27e51c5cd526-tls-certs\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.615790 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.607641 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-dshm\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.625760 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.625723 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9cm\" (UniqueName: \"kubernetes.io/projected/176dd3a2-8222-4882-97d7-27e51c5cd526-kube-api-access-4t9cm\") pod \"simulator-w6-517149-kserve-56b657fc78-xscq5\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.672821 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.672779 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:36.980085 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:36.980052 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5"] Nov 03 21:42:36.981009 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:42:36.980979 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod176dd3a2_8222_4882_97d7_27e51c5cd526.slice/crio-29a0a99d38b1c01aa4cfb097418b7f9265607ab145079f97e4c451bb3c7924c9 WatchSource:0}: Error finding container 29a0a99d38b1c01aa4cfb097418b7f9265607ab145079f97e4c451bb3c7924c9: Status 404 returned error can't find the container with id 29a0a99d38b1c01aa4cfb097418b7f9265607ab145079f97e4c451bb3c7924c9 Nov 03 21:42:37.968749 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:37.968704 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" event={"ID":"176dd3a2-8222-4882-97d7-27e51c5cd526","Type":"ContainerStarted","Data":"ab205213a85bcef421c895b93b41447eb7df71dbe0fff7d15d99736e154adc51"} Nov 03 21:42:37.969235 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:37.968760 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" event={"ID":"176dd3a2-8222-4882-97d7-27e51c5cd526","Type":"ContainerStarted","Data":"29a0a99d38b1c01aa4cfb097418b7f9265607ab145079f97e4c451bb3c7924c9"} Nov 03 21:42:38.010521 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:38.010465 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podStartSLOduration=2.010451029 podStartE2EDuration="2.010451029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:38.008636383 +0000 UTC m=+773.515864682" watchObservedRunningTime="2026-11-03 21:42:38.010451029 +0000 UTC m=+773.517679323" Nov 03 21:42:38.673555 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:38.673518 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:38.958174 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:38.958134 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:38.971615 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:38.971588 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:38.987347 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:38.987318 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:42:40.786363 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:42:40.786323 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:42:46.673697 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:46.673646 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:46.687246 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:46.687213 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:47.056237 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:47.056205 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:42:48.608970 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.608930 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h"] Nov 03 21:42:48.613203 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.613177 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.634173 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.634138 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h"] Nov 03 21:42:48.644286 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.644252 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-27fdf4\"/\"kube-root-ca.crt\"" Nov 03 21:42:48.644546 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.644287 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-27fdf4\"/\"openshift-service-ca.crt\"" Nov 03 21:42:48.644546 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.644407 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-27fdf4\"/\"default-dockercfg-xk82w\"" Nov 03 21:42:48.644766 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.644538 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-27fdf4\"/\"premium-w2-27fdf4-kserve-self-signed-certs\"" Nov 03 21:42:48.655323 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.655281 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-dshm\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.655731 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.655631 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qncsx\" (UniqueName: \"kubernetes.io/projected/c00e0aa7-7eb9-473d-b499-c0c9d4715981-kube-api-access-qncsx\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.655894 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.655870 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-model-cache\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.656125 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.656103 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-home\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.656281 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.656255 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00e0aa7-7eb9-473d-b499-c0c9d4715981-tls-certs\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.656362 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.656294 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-tmp-dir\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.757850 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.757804 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-tmp-dir\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.757850 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.757846 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-dshm\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.758124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.757953 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qncsx\" (UniqueName: \"kubernetes.io/projected/c00e0aa7-7eb9-473d-b499-c0c9d4715981-kube-api-access-qncsx\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.758124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.758004 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-model-cache\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.758124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.758037 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-home\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.758124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.758062 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00e0aa7-7eb9-473d-b499-c0c9d4715981-tls-certs\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.758347 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.758305 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-tmp-dir\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.758422 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.758398 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-model-cache\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.758471 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.758451 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-home\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.760194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.760172 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-dshm\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.760463 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.760444 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00e0aa7-7eb9-473d-b499-c0c9d4715981-tls-certs\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.777994 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.777964 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qncsx\" (UniqueName: \"kubernetes.io/projected/c00e0aa7-7eb9-473d-b499-c0c9d4715981-kube-api-access-qncsx\") pod \"premium-w2-27fdf4-kserve-65bc566857-gbz7h\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:48.933924 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:48.933811 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:49.235363 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:49.235329 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h"] Nov 03 21:42:49.236816 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:42:49.236783 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc00e0aa7_7eb9_473d_b499_c0c9d4715981.slice/crio-b26ea0d43186511821e59c3764798b8c1bedcf0b3326c2e242ff76a9b3fd8837 WatchSource:0}: Error finding container b26ea0d43186511821e59c3764798b8c1bedcf0b3326c2e242ff76a9b3fd8837: Status 404 returned error can't find the container with id b26ea0d43186511821e59c3764798b8c1bedcf0b3326c2e242ff76a9b3fd8837 Nov 03 21:42:50.067324 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:50.067290 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" event={"ID":"c00e0aa7-7eb9-473d-b499-c0c9d4715981","Type":"ContainerStarted","Data":"1313d2f7abd621f5c2fccb8cf0e1bf6f2c22dcb52d5a984db947e85fab36cfa3"} Nov 03 21:42:50.067324 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:50.067336 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" event={"ID":"c00e0aa7-7eb9-473d-b499-c0c9d4715981","Type":"ContainerStarted","Data":"b26ea0d43186511821e59c3764798b8c1bedcf0b3326c2e242ff76a9b3fd8837"} Nov 03 21:42:50.126684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:50.109935 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podStartSLOduration=2.109912258 podStartE2EDuration="2.109912258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:50.107748469 +0000 UTC m=+785.614976823" watchObservedRunningTime="2026-11-03 21:42:50.109912258 +0000 UTC m=+785.617140538" Nov 03 21:42:50.934350 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:50.934306 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:58.934727 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:58.934686 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:58.948077 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:58.948048 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:42:59.145010 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:42:59.144966 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:43:08.330126 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.330086 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk"] Nov 03 21:43:08.334395 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.334367 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.351403 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.351374 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk"] Nov 03 21:43:08.360036 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.360013 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd5620\"/\"premium-w3-dd5620-kserve-self-signed-certs\"" Nov 03 21:43:08.519596 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.519350 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s45m7\" (UniqueName: \"kubernetes.io/projected/f09eb21d-9777-437b-a2a1-efce30d9d145-kube-api-access-s45m7\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.519596 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.519428 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-model-cache\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.519596 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.519483 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09eb21d-9777-437b-a2a1-efce30d9d145-tls-certs\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.519596 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.519564 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-home\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.519975 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.519619 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-tmp-dir\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.519975 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.519654 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-dshm\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.621044 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.620960 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s45m7\" (UniqueName: \"kubernetes.io/projected/f09eb21d-9777-437b-a2a1-efce30d9d145-kube-api-access-s45m7\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.621044 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.621014 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-model-cache\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.621275 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.621058 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09eb21d-9777-437b-a2a1-efce30d9d145-tls-certs\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.621275 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.621116 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-home\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.621275 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.621165 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-tmp-dir\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.621275 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.621202 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-dshm\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.621514 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.621492 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-model-cache\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.621579 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.621552 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-home\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.621638 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.621616 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-tmp-dir\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.623552 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.623529 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-dshm\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.623727 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.623703 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09eb21d-9777-437b-a2a1-efce30d9d145-tls-certs\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.640907 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.640884 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s45m7\" (UniqueName: \"kubernetes.io/projected/f09eb21d-9777-437b-a2a1-efce30d9d145-kube-api-access-s45m7\") pod \"premium-w3-dd5620-kserve-85bfdb5f88-wbgmk\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.651801 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.651776 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:08.953773 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:08.953746 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk"] Nov 03 21:43:08.955241 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:43:08.955212 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf09eb21d_9777_437b_a2a1_efce30d9d145.slice/crio-aacc20229b9a72889cfbe394b2b0262774f7af85ca93b3caa9f23717718edaf1 WatchSource:0}: Error finding container aacc20229b9a72889cfbe394b2b0262774f7af85ca93b3caa9f23717718edaf1: Status 404 returned error can't find the container with id aacc20229b9a72889cfbe394b2b0262774f7af85ca93b3caa9f23717718edaf1 Nov 03 21:43:09.202413 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:09.202317 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" event={"ID":"f09eb21d-9777-437b-a2a1-efce30d9d145","Type":"ContainerStarted","Data":"719b99d06895604c9a764bb33269df18fbbda2ca7e372df484501a7cdee215d6"} Nov 03 21:43:09.202413 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:09.202360 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" event={"ID":"f09eb21d-9777-437b-a2a1-efce30d9d145","Type":"ContainerStarted","Data":"aacc20229b9a72889cfbe394b2b0262774f7af85ca93b3caa9f23717718edaf1"} Nov 03 21:43:09.652959 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:09.652879 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:10.777436 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:43:10.777402 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:43:18.652276 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:18.652237 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:18.665305 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:18.665278 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:18.706615 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:18.706568 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podStartSLOduration=10.706551935 podStartE2EDuration="10.706551935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:09.242319002 +0000 UTC m=+804.749547307" watchObservedRunningTime="2026-11-03 21:43:18.706551935 +0000 UTC m=+814.213780215" Nov 03 21:43:19.317504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:19.317473 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:43:23.814415 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:23.814376 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz"] Nov 03 21:43:23.819616 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:23.819588 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:23.844995 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:23.844952 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz"] Nov 03 21:43:23.856289 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:23.856253 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-517149\"/\"premium-w6-517149-kserve-self-signed-certs\"" Nov 03 21:43:23.927705 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:23.927632 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0208e182-3db8-43cc-a945-9cea424a6796-tls-certs\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:23.927885 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:23.927855 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-model-cache\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:23.927930 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:23.927900 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nf95\" (UniqueName: \"kubernetes.io/projected/0208e182-3db8-43cc-a945-9cea424a6796-kube-api-access-7nf95\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:23.928062 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:23.928046 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-tmp-dir\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:23.928144 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:23.928130 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-home\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:23.928186 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:23.928170 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-dshm\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:24.029474 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:24.029430 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0208e182-3db8-43cc-a945-9cea424a6796-tls-certs\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:24.029893 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:24.029872 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-model-cache\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:24.030463 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:24.030439 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nf95\" (UniqueName: \"kubernetes.io/projected/0208e182-3db8-43cc-a945-9cea424a6796-kube-api-access-7nf95\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:24.030727 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:24.030709 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-tmp-dir\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:24.030917 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:24.030899 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-home\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:24.031026 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:24.031012 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-dshm\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:24.031848 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:24.030393 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-model-cache\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:24.032189 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:24.032160 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-home\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:24.032326 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:24.032282 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-tmp-dir\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:24.035224 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:24.035196 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-dshm\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:24.038163 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:24.038136 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0208e182-3db8-43cc-a945-9cea424a6796-tls-certs\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:24.054412 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:24.054379 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nf95\" (UniqueName: \"kubernetes.io/projected/0208e182-3db8-43cc-a945-9cea424a6796-kube-api-access-7nf95\") pod \"premium-w6-517149-kserve-69fb88479-zwdcz\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:24.150127 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:24.150029 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:24.447635 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:24.447599 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz"] Nov 03 21:43:24.448765 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:43:24.448735 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0208e182_3db8_43cc_a945_9cea424a6796.slice/crio-cf69f419533b3386337d70c999bfd2dca9e18049b139f8c4e718d8009b837f65 WatchSource:0}: Error finding container cf69f419533b3386337d70c999bfd2dca9e18049b139f8c4e718d8009b837f65: Status 404 returned error can't find the container with id cf69f419533b3386337d70c999bfd2dca9e18049b139f8c4e718d8009b837f65 Nov 03 21:43:25.355522 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:25.355475 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" event={"ID":"0208e182-3db8-43cc-a945-9cea424a6796","Type":"ContainerStarted","Data":"ca81f569e3516dba457a5eacb9e28a971536e990d82c4510b41a62077f3025cf"} Nov 03 21:43:25.356000 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:25.355531 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" event={"ID":"0208e182-3db8-43cc-a945-9cea424a6796","Type":"ContainerStarted","Data":"cf69f419533b3386337d70c999bfd2dca9e18049b139f8c4e718d8009b837f65"} Nov 03 21:43:25.400325 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:25.400264 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podStartSLOduration=2.400249902 podStartE2EDuration="2.400249902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:25.398054817 +0000 UTC m=+820.905283097" watchObservedRunningTime="2026-11-03 21:43:25.400249902 +0000 UTC m=+820.907478182" Nov 03 21:43:26.151050 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:26.151011 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:34.150640 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:34.150600 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:34.165869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:34.165839 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:34.430137 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:34.430058 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:43:39.790013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.789979 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l"] Nov 03 21:43:39.793066 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.793050 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.808934 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.808909 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-4eced5-openshift-default-dockercfg-pf9dz\"" Nov 03 21:43:39.827604 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.827571 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l"] Nov 03 21:43:39.831239 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.831210 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/557fc6e5-610e-457d-895c-fa9746131887-istiod-ca-cert\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.831388 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.831262 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-workload-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.831464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.831437 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpl6j\" (UniqueName: \"kubernetes.io/projected/557fc6e5-610e-457d-895c-fa9746131887-kube-api-access-cpl6j\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.831523 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.831481 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-credential-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.831594 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.831572 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-istio-envoy\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.831655 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.831637 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/557fc6e5-610e-457d-895c-fa9746131887-istio-podinfo\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.831983 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.831782 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-istio-data\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.832494 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.832468 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-workload-certs\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.841389 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.841346 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/557fc6e5-610e-457d-895c-fa9746131887-istio-token\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.943026 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.942984 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-istio-data\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.943722 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.943688 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-istio-data\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.943849 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.943701 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-workload-certs\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.943849 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.943837 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/557fc6e5-610e-457d-895c-fa9746131887-istio-token\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.943931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.943894 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/557fc6e5-610e-457d-895c-fa9746131887-istiod-ca-cert\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.944013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.943929 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-workload-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.944055 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.944017 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpl6j\" (UniqueName: \"kubernetes.io/projected/557fc6e5-610e-457d-895c-fa9746131887-kube-api-access-cpl6j\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.944191 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.944068 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-workload-certs\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.944191 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.944081 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-credential-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.944191 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.944138 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-istio-envoy\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.944385 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.944196 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/557fc6e5-610e-457d-895c-fa9746131887-istio-podinfo\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.944527 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.944505 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-credential-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.944887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.944862 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-workload-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.945358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.945335 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/557fc6e5-610e-457d-895c-fa9746131887-istiod-ca-cert\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.947183 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.947157 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-istio-envoy\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.947615 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.947593 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/557fc6e5-610e-457d-895c-fa9746131887-istio-podinfo\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.966473 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.966448 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/557fc6e5-610e-457d-895c-fa9746131887-istio-token\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:39.966644 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:39.966625 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpl6j\" (UniqueName: \"kubernetes.io/projected/557fc6e5-610e-457d-895c-fa9746131887-kube-api-access-cpl6j\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:40.106847 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:40.106759 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:40.426021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:40.425802 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l"] Nov 03 21:43:40.426757 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:43:40.426733 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557fc6e5_610e_457d_895c_fa9746131887.slice/crio-6e9eaf7d0467111f11e92df9d4cd2693c40d7d9e927cc284438f161b64250acd WatchSource:0}: Error finding container 6e9eaf7d0467111f11e92df9d4cd2693c40d7d9e927cc284438f161b64250acd: Status 404 returned error can't find the container with id 6e9eaf7d0467111f11e92df9d4cd2693c40d7d9e927cc284438f161b64250acd Nov 03 21:43:40.428754 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:40.428718 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:43:40.428855 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:40.428799 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:43:40.428855 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:40.428847 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:43:40.486585 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:40.486559 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" event={"ID":"557fc6e5-610e-457d-895c-fa9746131887","Type":"ContainerStarted","Data":"6e9eaf7d0467111f11e92df9d4cd2693c40d7d9e927cc284438f161b64250acd"} Nov 03 21:43:40.782425 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:43:40.782383 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:43:41.495260 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:41.495223 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" event={"ID":"557fc6e5-610e-457d-895c-fa9746131887","Type":"ContainerStarted","Data":"f1e60ac9e83749d489d29f78235362e03bad2f1acbc4e53a44331fbec92a6f1b"} Nov 03 21:43:41.556582 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:41.556519 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" podStartSLOduration=2.556503041 podStartE2EDuration="2.556503041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:41.553784952 +0000 UTC m=+837.061013234" watchObservedRunningTime="2026-11-03 21:43:41.556503041 +0000 UTC m=+837.063731321" Nov 03 21:43:42.129687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:42.112327 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:42.129687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:42.117765 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:43:42.129687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:42.117821 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" podUID="557fc6e5-610e-457d-895c-fa9746131887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 21:43:43.107704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:43.107633 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:43:43.108219 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:43.107735 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" podUID="557fc6e5-610e-457d-895c-fa9746131887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 21:43:44.112686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:44.107684 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:43:44.112686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:44.107748 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" podUID="557fc6e5-610e-457d-895c-fa9746131887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 21:43:45.107336 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.107303 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:43:45.107515 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.107357 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" podUID="557fc6e5-610e-457d-895c-fa9746131887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 21:43:45.343483 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.339943 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh"] Nov 03 21:43:45.351363 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.351333 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh"] Nov 03 21:43:45.351608 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.351491 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" Nov 03 21:43:45.367753 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.367729 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:43:45.381508 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.381485 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:43:45.441199 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.441166 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de348e53-4616-4e56-9fde-954cff5e5f89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1e57fd5-fssrh\" (UID: \"de348e53-4616-4e56-9fde-954cff5e5f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" Nov 03 21:43:45.441492 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.441470 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-292st\" (UniqueName: \"kubernetes.io/projected/de348e53-4616-4e56-9fde-954cff5e5f89-kube-api-access-292st\") pod \"maas-api-revoke-sub-e1e57fd5-fssrh\" (UID: \"de348e53-4616-4e56-9fde-954cff5e5f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" Nov 03 21:43:45.441682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.441653 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de348e53-4616-4e56-9fde-954cff5e5f89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1e57fd5-fssrh\" (UID: \"de348e53-4616-4e56-9fde-954cff5e5f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" Nov 03 21:43:45.543572 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.543540 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de348e53-4616-4e56-9fde-954cff5e5f89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1e57fd5-fssrh\" (UID: \"de348e53-4616-4e56-9fde-954cff5e5f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" Nov 03 21:43:45.543770 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.543603 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-292st\" (UniqueName: \"kubernetes.io/projected/de348e53-4616-4e56-9fde-954cff5e5f89-kube-api-access-292st\") pod \"maas-api-revoke-sub-e1e57fd5-fssrh\" (UID: \"de348e53-4616-4e56-9fde-954cff5e5f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" Nov 03 21:43:45.543770 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.543710 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de348e53-4616-4e56-9fde-954cff5e5f89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1e57fd5-fssrh\" (UID: \"de348e53-4616-4e56-9fde-954cff5e5f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" Nov 03 21:43:45.544327 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.544302 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de348e53-4616-4e56-9fde-954cff5e5f89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1e57fd5-fssrh\" (UID: \"de348e53-4616-4e56-9fde-954cff5e5f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" Nov 03 21:43:45.546357 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.546187 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de348e53-4616-4e56-9fde-954cff5e5f89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1e57fd5-fssrh\" (UID: \"de348e53-4616-4e56-9fde-954cff5e5f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" Nov 03 21:43:45.564524 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.564501 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-292st\" (UniqueName: \"kubernetes.io/projected/de348e53-4616-4e56-9fde-954cff5e5f89-kube-api-access-292st\") pod \"maas-api-revoke-sub-e1e57fd5-fssrh\" (UID: \"de348e53-4616-4e56-9fde-954cff5e5f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" Nov 03 21:43:45.669310 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:45.669199 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" Nov 03 21:43:46.000246 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:46.000215 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh"] Nov 03 21:43:46.001111 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:43:46.001087 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde348e53_4616_4e56_9fde_954cff5e5f89.slice/crio-9d8aa330c72b0b9d1456fc82d0849da97f3e40a02e8cf9786c65648dda68542b WatchSource:0}: Error finding container 9d8aa330c72b0b9d1456fc82d0849da97f3e40a02e8cf9786c65648dda68542b: Status 404 returned error can't find the container with id 9d8aa330c72b0b9d1456fc82d0849da97f3e40a02e8cf9786c65648dda68542b Nov 03 21:43:46.111687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:46.108003 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:43:46.111687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:46.108059 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" podUID="557fc6e5-610e-457d-895c-fa9746131887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 21:43:46.552003 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:46.551963 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" event={"ID":"de348e53-4616-4e56-9fde-954cff5e5f89","Type":"ContainerStarted","Data":"9d8aa330c72b0b9d1456fc82d0849da97f3e40a02e8cf9786c65648dda68542b"} Nov 03 21:43:47.358971 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:47.358939 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:47.359734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:47.359706 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:47.360775 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:47.360750 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:43:50.717866 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:50.717825 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w"] Nov 03 21:43:50.722502 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:50.722479 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" Nov 03 21:43:50.739760 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:50.737912 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w"] Nov 03 21:43:50.914960 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:50.914927 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79ae8406-bmg9w\" (UID: \"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" Nov 03 21:43:50.914960 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:50.914966 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4cmp\" (UniqueName: \"kubernetes.io/projected/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-kube-api-access-x4cmp\") pod \"maas-api-revoke-sub-79ae8406-bmg9w\" (UID: \"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" Nov 03 21:43:50.915188 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:50.915099 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79ae8406-bmg9w\" (UID: \"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" Nov 03 21:43:51.016048 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:51.015961 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79ae8406-bmg9w\" (UID: \"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" Nov 03 21:43:51.016201 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:51.016181 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79ae8406-bmg9w\" (UID: \"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" Nov 03 21:43:51.016282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:51.016212 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4cmp\" (UniqueName: \"kubernetes.io/projected/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-kube-api-access-x4cmp\") pod \"maas-api-revoke-sub-79ae8406-bmg9w\" (UID: \"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" Nov 03 21:43:51.017007 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:51.016981 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79ae8406-bmg9w\" (UID: \"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" Nov 03 21:43:51.018271 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:51.018247 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79ae8406-bmg9w\" (UID: \"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" Nov 03 21:43:51.036088 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:51.036058 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4cmp\" (UniqueName: \"kubernetes.io/projected/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-kube-api-access-x4cmp\") pod \"maas-api-revoke-sub-79ae8406-bmg9w\" (UID: \"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" Nov 03 21:43:51.073699 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:51.073655 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" Nov 03 21:43:51.373155 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:51.373128 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w"] Nov 03 21:43:51.373544 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:43:51.373515 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4e40f3_c0f0_4be2_a5e6_2d94850e0a81.slice/crio-f9a4564c4f39e6603699f824c9ebaedd0750d73b773a1646c0fa081fb2f33b22 WatchSource:0}: Error finding container f9a4564c4f39e6603699f824c9ebaedd0750d73b773a1646c0fa081fb2f33b22: Status 404 returned error can't find the container with id f9a4564c4f39e6603699f824c9ebaedd0750d73b773a1646c0fa081fb2f33b22 Nov 03 21:43:51.588874 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:51.588838 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" event={"ID":"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81","Type":"ContainerStarted","Data":"f9a4564c4f39e6603699f824c9ebaedd0750d73b773a1646c0fa081fb2f33b22"} Nov 03 21:43:54.685762 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:54.685725 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" event={"ID":"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81","Type":"ContainerStarted","Data":"835082d3140ca2344d8107ac398707486793123fdba3dad55029855519e1adf9"} Nov 03 21:43:54.736598 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:54.736520 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" podStartSLOduration=2.39994529 podStartE2EDuration="4.736500264s" totalImagesPullingTime="2.336554974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:54.730103986 +0000 UTC m=+850.237332266" watchObservedRunningTime="2026-11-03 21:43:54.736500264 +0000 UTC m=+850.243728543" Nov 03 21:43:54.886932 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:54.886895 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg"] Nov 03 21:43:54.894953 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:54.894924 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:54.901523 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:54.901481 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg"] Nov 03 21:43:54.953742 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:54.943960 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e246f9-openshift-default-dockercfg-vn7bt\"" Nov 03 21:43:55.097585 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.077632 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-data\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.097585 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.077724 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-workload-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.097585 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.077786 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjlch\" (UniqueName: \"kubernetes.io/projected/ca8a2829-0a62-49af-b6ba-9546aa670350-kube-api-access-wjlch\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.097585 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.077845 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-envoy\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.097585 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.077874 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-podinfo\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.097585 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.077927 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-token\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.097585 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.077990 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-credential-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.097585 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.078040 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8a2829-0a62-49af-b6ba-9546aa670350-istiod-ca-cert\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.097585 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.078145 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-workload-certs\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.179357 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.179324 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-token\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.179563 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.179381 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-credential-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.179563 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.179416 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8a2829-0a62-49af-b6ba-9546aa670350-istiod-ca-cert\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.179728 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.179608 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-workload-certs\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.179728 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.179691 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-data\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.179828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.179746 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-workload-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.179828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.179791 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-credential-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.179828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.179807 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjlch\" (UniqueName: \"kubernetes.io/projected/ca8a2829-0a62-49af-b6ba-9546aa670350-kube-api-access-wjlch\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.179981 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.179868 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-envoy\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.179981 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.179892 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-podinfo\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.180099 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.180008 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-workload-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.180099 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.180027 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-workload-certs\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.180197 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.180129 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-data\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.180260 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.180239 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8a2829-0a62-49af-b6ba-9546aa670350-istiod-ca-cert\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.182294 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.182269 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-envoy\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.182471 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.182453 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-podinfo\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.200943 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.200917 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-token\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.201220 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.201203 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjlch\" (UniqueName: \"kubernetes.io/projected/ca8a2829-0a62-49af-b6ba-9546aa670350-kube-api-access-wjlch\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.402861 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.402777 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:55.693325 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.693235 5041 generic.go:356] "Generic (PLEG): container finished" podID="ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81" containerID="835082d3140ca2344d8107ac398707486793123fdba3dad55029855519e1adf9" exitCode=0 Nov 03 21:43:55.693325 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.693311 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" event={"ID":"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81","Type":"ContainerDied","Data":"835082d3140ca2344d8107ac398707486793123fdba3dad55029855519e1adf9"} Nov 03 21:43:55.716624 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:43:55.716596 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8a2829_0a62_49af_b6ba_9546aa670350.slice/crio-f9fbfc4d138d22741bcf8d880e728fdc99bcce818ec1a909923e4b685cebf7f2 WatchSource:0}: Error finding container f9fbfc4d138d22741bcf8d880e728fdc99bcce818ec1a909923e4b685cebf7f2: Status 404 returned error can't find the container with id f9fbfc4d138d22741bcf8d880e728fdc99bcce818ec1a909923e4b685cebf7f2 Nov 03 21:43:55.719142 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.719112 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:43:55.719240 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.719176 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:43:55.719240 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.719203 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:43:55.729246 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:55.729219 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg"] Nov 03 21:43:56.701025 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:56.700985 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" event={"ID":"ca8a2829-0a62-49af-b6ba-9546aa670350","Type":"ContainerStarted","Data":"95803f3ba7efe83abc13649fce4b12272b2ecee70ac4facc71dd570b1a33151c"} Nov 03 21:43:56.701449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:56.701051 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" event={"ID":"ca8a2829-0a62-49af-b6ba-9546aa670350","Type":"ContainerStarted","Data":"f9fbfc4d138d22741bcf8d880e728fdc99bcce818ec1a909923e4b685cebf7f2"} Nov 03 21:43:56.766702 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:56.766617 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" podStartSLOduration=2.766602073 podStartE2EDuration="2.766602073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:56.765719773 +0000 UTC m=+852.272948065" watchObservedRunningTime="2026-11-03 21:43:56.766602073 +0000 UTC m=+852.273830353" Nov 03 21:43:56.954336 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:56.954269 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" Nov 03 21:43:57.005950 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:57.005921 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-maas-api-openshift-ca\") pod \"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81\" (UID: \"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81\") " Nov 03 21:43:57.006107 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:57.005959 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-maas-api-xks-ca\") pod \"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81\" (UID: \"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81\") " Nov 03 21:43:57.006107 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:57.006094 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-kube-api-access-x4cmp\" (UniqueName: \"kubernetes.io/projected/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-kube-api-access-x4cmp\") pod \"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81\" (UID: \"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81\") " Nov 03 21:43:57.006267 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:57.006241 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-maas-api-openshift-ca" pod "ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81" (UID: "ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:43:57.006554 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:57.006530 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:43:57.053948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:57.053889 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-maas-api-xks-ca" pod "ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81" (UID: "ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:43:57.054251 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:57.054187 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-kube-api-access-x4cmp" pod "ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81" (UID: "ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81"). InnerVolumeSpecName "kube-api-access-x4cmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:43:57.126488 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:57.126413 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:43:57.126488 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:57.126458 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4cmp\" (UniqueName: \"kubernetes.io/projected/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81-kube-api-access-x4cmp\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:43:57.403468 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:57.403378 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:43:57.404827 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:57.404801 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:43:57.404933 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:57.404854 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" podUID="ca8a2829-0a62-49af-b6ba-9546aa670350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 21:43:57.715689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:57.713021 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" Nov 03 21:43:57.726814 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:57.726717 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w" event={"ID":"ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81","Type":"ContainerDied","Data":"f9a4564c4f39e6603699f824c9ebaedd0750d73b773a1646c0fa081fb2f33b22"} Nov 03 21:43:57.726814 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:57.726783 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9a4564c4f39e6603699f824c9ebaedd0750d73b773a1646c0fa081fb2f33b22" Nov 03 21:43:58.041705 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.041587 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj"] Nov 03 21:43:58.042292 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.042269 5041 cpu_manager.go:414] "containerMap: removing container" podUID="ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81" containerName="revoke-keys" Nov 03 21:43:58.042292 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.042292 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81" containerName="revoke-keys" Nov 03 21:43:58.042490 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.042477 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81" containerName="revoke-keys" Nov 03 21:43:58.046128 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.046106 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" Nov 03 21:43:58.069474 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.069441 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj"] Nov 03 21:43:58.151825 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.151791 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d3f6fe1-flwcj\" (UID: \"6b7011cf-b114-4f2b-9ff5-f2072021c2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" Nov 03 21:43:58.152010 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.151928 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d3f6fe1-flwcj\" (UID: \"6b7011cf-b114-4f2b-9ff5-f2072021c2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" Nov 03 21:43:58.152010 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.151963 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgmz2\" (UniqueName: \"kubernetes.io/projected/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-kube-api-access-tgmz2\") pod \"maas-api-revoke-sub-1d3f6fe1-flwcj\" (UID: \"6b7011cf-b114-4f2b-9ff5-f2072021c2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" Nov 03 21:43:58.252838 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.252802 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d3f6fe1-flwcj\" (UID: \"6b7011cf-b114-4f2b-9ff5-f2072021c2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" Nov 03 21:43:58.253037 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.252890 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d3f6fe1-flwcj\" (UID: \"6b7011cf-b114-4f2b-9ff5-f2072021c2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" Nov 03 21:43:58.253037 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.252922 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgmz2\" (UniqueName: \"kubernetes.io/projected/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-kube-api-access-tgmz2\") pod \"maas-api-revoke-sub-1d3f6fe1-flwcj\" (UID: \"6b7011cf-b114-4f2b-9ff5-f2072021c2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" Nov 03 21:43:58.253642 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.253611 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d3f6fe1-flwcj\" (UID: \"6b7011cf-b114-4f2b-9ff5-f2072021c2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" Nov 03 21:43:58.255289 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.255266 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d3f6fe1-flwcj\" (UID: \"6b7011cf-b114-4f2b-9ff5-f2072021c2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" Nov 03 21:43:58.273085 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.273055 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgmz2\" (UniqueName: \"kubernetes.io/projected/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-kube-api-access-tgmz2\") pod \"maas-api-revoke-sub-1d3f6fe1-flwcj\" (UID: \"6b7011cf-b114-4f2b-9ff5-f2072021c2b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" Nov 03 21:43:58.372414 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.372330 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" Nov 03 21:43:58.403830 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.403792 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:43:58.403937 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.403866 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" podUID="ca8a2829-0a62-49af-b6ba-9546aa670350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 21:43:58.724050 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:58.724007 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj"] Nov 03 21:43:59.403518 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:59.403486 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:43:59.403712 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:59.403537 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" podUID="ca8a2829-0a62-49af-b6ba-9546aa670350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 21:43:59.754490 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:59.754456 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" event={"ID":"de348e53-4616-4e56-9fde-954cff5e5f89","Type":"ContainerStarted","Data":"d24de2e036e77e0d63e860318e93c9bc66b96def4a2fddd1c486ec6ebbd43f53"} Nov 03 21:43:59.756255 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:59.756229 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" event={"ID":"6b7011cf-b114-4f2b-9ff5-f2072021c2b2","Type":"ContainerStarted","Data":"9c7eec835f12b2a6327e3be182622c8c87ef06040ee9c56906ddbd953105cfcb"} Nov 03 21:43:59.756391 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:59.756273 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" event={"ID":"6b7011cf-b114-4f2b-9ff5-f2072021c2b2","Type":"ContainerStarted","Data":"5cc11e3d878f84bc8f1bc508d23d74057dab51948fca69006e215058528882d5"} Nov 03 21:43:59.796302 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:59.796243 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" podStartSLOduration=1.55793969 podStartE2EDuration="14.796229807s" totalImagesPullingTime="13.238290117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:59.794871139 +0000 UTC m=+855.302099413" watchObservedRunningTime="2026-11-03 21:43:59.796229807 +0000 UTC m=+855.303458087" Nov 03 21:43:59.841906 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:43:59.841871 5041 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7011cf_b114_4f2b_9ff5_f2072021c2b2.slice/crio-9c7eec835f12b2a6327e3be182622c8c87ef06040ee9c56906ddbd953105cfcb.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:43:59.849244 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:43:59.849191 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" podStartSLOduration=1.849172638 podStartE2EDuration="1.849172638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:43:59.846997128 +0000 UTC m=+855.354225408" watchObservedRunningTime="2026-11-03 21:43:59.849172638 +0000 UTC m=+855.356400918" Nov 03 21:44:00.403417 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:00.403370 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:44:00.403611 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:00.403425 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" podUID="ca8a2829-0a62-49af-b6ba-9546aa670350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 21:44:00.765526 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:00.765486 5041 generic.go:356] "Generic (PLEG): container finished" podID="de348e53-4616-4e56-9fde-954cff5e5f89" containerID="d24de2e036e77e0d63e860318e93c9bc66b96def4a2fddd1c486ec6ebbd43f53" exitCode=0 Nov 03 21:44:00.765526 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:00.765517 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" event={"ID":"de348e53-4616-4e56-9fde-954cff5e5f89","Type":"ContainerDied","Data":"d24de2e036e77e0d63e860318e93c9bc66b96def4a2fddd1c486ec6ebbd43f53"} Nov 03 21:44:00.767476 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:00.767450 5041 generic.go:356] "Generic (PLEG): container finished" podID="6b7011cf-b114-4f2b-9ff5-f2072021c2b2" containerID="9c7eec835f12b2a6327e3be182622c8c87ef06040ee9c56906ddbd953105cfcb" exitCode=0 Nov 03 21:44:00.767598 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:00.767517 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" event={"ID":"6b7011cf-b114-4f2b-9ff5-f2072021c2b2","Type":"ContainerDied","Data":"9c7eec835f12b2a6327e3be182622c8c87ef06040ee9c56906ddbd953105cfcb"} Nov 03 21:44:01.403842 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:01.403805 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 21:44:01.403999 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:01.403859 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" podUID="ca8a2829-0a62-49af-b6ba-9546aa670350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 21:44:01.988850 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:01.988823 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" Nov 03 21:44:01.993325 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:01.993262 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" Nov 03 21:44:02.101726 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.093590 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-maas-api-openshift-ca\") pod \"6b7011cf-b114-4f2b-9ff5-f2072021c2b2\" (UID: \"6b7011cf-b114-4f2b-9ff5-f2072021c2b2\") " Nov 03 21:44:02.101726 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.093717 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de348e53-4616-4e56-9fde-954cff5e5f89-kube-api-access-292st\" (UniqueName: \"kubernetes.io/projected/de348e53-4616-4e56-9fde-954cff5e5f89-kube-api-access-292st\") pod \"de348e53-4616-4e56-9fde-954cff5e5f89\" (UID: \"de348e53-4616-4e56-9fde-954cff5e5f89\") " Nov 03 21:44:02.101726 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.093788 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-maas-api-xks-ca\") pod \"6b7011cf-b114-4f2b-9ff5-f2072021c2b2\" (UID: \"6b7011cf-b114-4f2b-9ff5-f2072021c2b2\") " Nov 03 21:44:02.101726 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.093922 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-kube-api-access-tgmz2\" (UniqueName: \"kubernetes.io/projected/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-kube-api-access-tgmz2\") pod \"6b7011cf-b114-4f2b-9ff5-f2072021c2b2\" (UID: \"6b7011cf-b114-4f2b-9ff5-f2072021c2b2\") " Nov 03 21:44:02.101726 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.093982 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de348e53-4616-4e56-9fde-954cff5e5f89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de348e53-4616-4e56-9fde-954cff5e5f89-maas-api-xks-ca\") pod \"de348e53-4616-4e56-9fde-954cff5e5f89\" (UID: \"de348e53-4616-4e56-9fde-954cff5e5f89\") " Nov 03 21:44:02.101726 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.093983 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-maas-api-openshift-ca" pod "6b7011cf-b114-4f2b-9ff5-f2072021c2b2" (UID: "6b7011cf-b114-4f2b-9ff5-f2072021c2b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:02.101726 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.094055 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de348e53-4616-4e56-9fde-954cff5e5f89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de348e53-4616-4e56-9fde-954cff5e5f89-maas-api-openshift-ca\") pod \"de348e53-4616-4e56-9fde-954cff5e5f89\" (UID: \"de348e53-4616-4e56-9fde-954cff5e5f89\") " Nov 03 21:44:02.101726 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.094540 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:02.101726 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.094911 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de348e53-4616-4e56-9fde-954cff5e5f89-maas-api-openshift-ca" pod "de348e53-4616-4e56-9fde-954cff5e5f89" (UID: "de348e53-4616-4e56-9fde-954cff5e5f89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:02.101726 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.097129 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-maas-api-xks-ca" pod "6b7011cf-b114-4f2b-9ff5-f2072021c2b2" (UID: "6b7011cf-b114-4f2b-9ff5-f2072021c2b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:02.101726 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.097872 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de348e53-4616-4e56-9fde-954cff5e5f89-maas-api-xks-ca" pod "de348e53-4616-4e56-9fde-954cff5e5f89" (UID: "de348e53-4616-4e56-9fde-954cff5e5f89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:02.101726 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.098571 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de348e53-4616-4e56-9fde-954cff5e5f89-kube-api-access-292st" pod "de348e53-4616-4e56-9fde-954cff5e5f89" (UID: "de348e53-4616-4e56-9fde-954cff5e5f89"). InnerVolumeSpecName "kube-api-access-292st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:02.101726 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.098758 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-kube-api-access-tgmz2" pod "6b7011cf-b114-4f2b-9ff5-f2072021c2b2" (UID: "6b7011cf-b114-4f2b-9ff5-f2072021c2b2"). InnerVolumeSpecName "kube-api-access-tgmz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:02.195632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.195594 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:02.195632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.195625 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgmz2\" (UniqueName: \"kubernetes.io/projected/6b7011cf-b114-4f2b-9ff5-f2072021c2b2-kube-api-access-tgmz2\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:02.195632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.195636 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de348e53-4616-4e56-9fde-954cff5e5f89-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:02.195632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.195645 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de348e53-4616-4e56-9fde-954cff5e5f89-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:02.196040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.195654 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-292st\" (UniqueName: \"kubernetes.io/projected/de348e53-4616-4e56-9fde-954cff5e5f89-kube-api-access-292st\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:02.464112 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.464077 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:44:02.465335 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.465308 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:44:02.466523 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.466493 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:44:02.785126 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.785040 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" event={"ID":"de348e53-4616-4e56-9fde-954cff5e5f89","Type":"ContainerDied","Data":"9d8aa330c72b0b9d1456fc82d0849da97f3e40a02e8cf9786c65648dda68542b"} Nov 03 21:44:02.785126 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.785104 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d8aa330c72b0b9d1456fc82d0849da97f3e40a02e8cf9786c65648dda68542b" Nov 03 21:44:02.785353 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.785207 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh" Nov 03 21:44:02.789554 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.789518 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" event={"ID":"6b7011cf-b114-4f2b-9ff5-f2072021c2b2","Type":"ContainerDied","Data":"5cc11e3d878f84bc8f1bc508d23d74057dab51948fca69006e215058528882d5"} Nov 03 21:44:02.789720 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.789595 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cc11e3d878f84bc8f1bc508d23d74057dab51948fca69006e215058528882d5" Nov 03 21:44:02.793770 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:02.793735 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj" Nov 03 21:44:05.050307 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.050276 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp"] Nov 03 21:44:05.050895 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.050877 5041 cpu_manager.go:414] "containerMap: removing container" podUID="6b7011cf-b114-4f2b-9ff5-f2072021c2b2" containerName="revoke-keys" Nov 03 21:44:05.050895 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.050896 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7011cf-b114-4f2b-9ff5-f2072021c2b2" containerName="revoke-keys" Nov 03 21:44:05.051064 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.050912 5041 cpu_manager.go:414] "containerMap: removing container" podUID="de348e53-4616-4e56-9fde-954cff5e5f89" containerName="revoke-keys" Nov 03 21:44:05.051064 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.050918 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de348e53-4616-4e56-9fde-954cff5e5f89" containerName="revoke-keys" Nov 03 21:44:05.051064 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.051037 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7011cf-b114-4f2b-9ff5-f2072021c2b2" containerName="revoke-keys" Nov 03 21:44:05.051064 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.051045 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="de348e53-4616-4e56-9fde-954cff5e5f89" containerName="revoke-keys" Nov 03 21:44:05.055561 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.055541 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp" Nov 03 21:44:05.072407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.072383 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp"] Nov 03 21:44:05.081345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.081323 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:44:05.081451 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.081392 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:44:05.129231 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.129199 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5498eaf4-5qtqp\" (UID: \"8ca4cd03-2b97-4cc8-9306-ea44a53e0668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp" Nov 03 21:44:05.129424 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.129329 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4v9r\" (UniqueName: \"kubernetes.io/projected/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-kube-api-access-h4v9r\") pod \"maas-api-revoke-sub-5498eaf4-5qtqp\" (UID: \"8ca4cd03-2b97-4cc8-9306-ea44a53e0668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp" Nov 03 21:44:05.129424 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.129394 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5498eaf4-5qtqp\" (UID: \"8ca4cd03-2b97-4cc8-9306-ea44a53e0668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp" Nov 03 21:44:05.231191 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.231159 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5498eaf4-5qtqp\" (UID: \"8ca4cd03-2b97-4cc8-9306-ea44a53e0668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp" Nov 03 21:44:05.231354 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.231212 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4v9r\" (UniqueName: \"kubernetes.io/projected/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-kube-api-access-h4v9r\") pod \"maas-api-revoke-sub-5498eaf4-5qtqp\" (UID: \"8ca4cd03-2b97-4cc8-9306-ea44a53e0668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp" Nov 03 21:44:05.231354 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.231236 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5498eaf4-5qtqp\" (UID: \"8ca4cd03-2b97-4cc8-9306-ea44a53e0668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp" Nov 03 21:44:05.231989 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.231968 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5498eaf4-5qtqp\" (UID: \"8ca4cd03-2b97-4cc8-9306-ea44a53e0668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp" Nov 03 21:44:05.233749 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.233727 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5498eaf4-5qtqp\" (UID: \"8ca4cd03-2b97-4cc8-9306-ea44a53e0668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp" Nov 03 21:44:05.251472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.251451 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4v9r\" (UniqueName: \"kubernetes.io/projected/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-kube-api-access-h4v9r\") pod \"maas-api-revoke-sub-5498eaf4-5qtqp\" (UID: \"8ca4cd03-2b97-4cc8-9306-ea44a53e0668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp" Nov 03 21:44:05.372708 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.372605 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp" Nov 03 21:44:05.672922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.672896 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp"] Nov 03 21:44:05.675038 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:44:05.675004 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca4cd03_2b97_4cc8_9306_ea44a53e0668.slice/crio-8a54994cb84a164b721993ddb353a6e583b6fc43721dcdd8745a88fc13eb4b02 WatchSource:0}: Error finding container 8a54994cb84a164b721993ddb353a6e583b6fc43721dcdd8745a88fc13eb4b02: Status 404 returned error can't find the container with id 8a54994cb84a164b721993ddb353a6e583b6fc43721dcdd8745a88fc13eb4b02 Nov 03 21:44:05.812138 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:05.812095 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp" event={"ID":"8ca4cd03-2b97-4cc8-9306-ea44a53e0668","Type":"ContainerStarted","Data":"8a54994cb84a164b721993ddb353a6e583b6fc43721dcdd8745a88fc13eb4b02"} Nov 03 21:44:06.759073 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:06.759039 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8"] Nov 03 21:44:06.763322 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:06.763301 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" Nov 03 21:44:06.781723 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:06.781623 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8"] Nov 03 21:44:06.822623 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:06.822589 5041 generic.go:356] "Generic (PLEG): container finished" podID="8ca4cd03-2b97-4cc8-9306-ea44a53e0668" containerID="5d1c21f6ba45de8a2edb87db00ebabea1862479701c06ff8f21ab8b1490d395b" exitCode=0 Nov 03 21:44:06.822832 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:06.822716 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp" event={"ID":"8ca4cd03-2b97-4cc8-9306-ea44a53e0668","Type":"ContainerDied","Data":"5d1c21f6ba45de8a2edb87db00ebabea1862479701c06ff8f21ab8b1490d395b"} Nov 03 21:44:06.851183 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:06.851148 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-maas-api-xks-ca\") pod \"maas-api-revoke-sub-288fc996-b9dm8\" (UID: \"b1ab6e34-3152-4ece-a98e-4fc7f59ec674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" Nov 03 21:44:06.851364 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:06.851300 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-288fc996-b9dm8\" (UID: \"b1ab6e34-3152-4ece-a98e-4fc7f59ec674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" Nov 03 21:44:06.851465 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:06.851448 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zngx\" (UniqueName: \"kubernetes.io/projected/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-kube-api-access-9zngx\") pod \"maas-api-revoke-sub-288fc996-b9dm8\" (UID: \"b1ab6e34-3152-4ece-a98e-4fc7f59ec674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" Nov 03 21:44:06.952266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:06.952228 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-maas-api-xks-ca\") pod \"maas-api-revoke-sub-288fc996-b9dm8\" (UID: \"b1ab6e34-3152-4ece-a98e-4fc7f59ec674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" Nov 03 21:44:06.952472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:06.952293 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-288fc996-b9dm8\" (UID: \"b1ab6e34-3152-4ece-a98e-4fc7f59ec674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" Nov 03 21:44:06.952472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:06.952352 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zngx\" (UniqueName: \"kubernetes.io/projected/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-kube-api-access-9zngx\") pod \"maas-api-revoke-sub-288fc996-b9dm8\" (UID: \"b1ab6e34-3152-4ece-a98e-4fc7f59ec674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" Nov 03 21:44:06.953095 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:06.953065 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-288fc996-b9dm8\" (UID: \"b1ab6e34-3152-4ece-a98e-4fc7f59ec674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" Nov 03 21:44:06.954604 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:06.954579 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-maas-api-xks-ca\") pod \"maas-api-revoke-sub-288fc996-b9dm8\" (UID: \"b1ab6e34-3152-4ece-a98e-4fc7f59ec674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" Nov 03 21:44:06.972779 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:06.972747 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zngx\" (UniqueName: \"kubernetes.io/projected/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-kube-api-access-9zngx\") pod \"maas-api-revoke-sub-288fc996-b9dm8\" (UID: \"b1ab6e34-3152-4ece-a98e-4fc7f59ec674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" Nov 03 21:44:07.079320 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:07.079225 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" Nov 03 21:44:07.390748 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:07.390705 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8"] Nov 03 21:44:07.390934 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:44:07.390911 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ab6e34_3152_4ece_a98e_4fc7f59ec674.slice/crio-33a46c4e74aad69d2dee37a706223e65f0f1815d412ea51d71616d5a714d88b2 WatchSource:0}: Error finding container 33a46c4e74aad69d2dee37a706223e65f0f1815d412ea51d71616d5a714d88b2: Status 404 returned error can't find the container with id 33a46c4e74aad69d2dee37a706223e65f0f1815d412ea51d71616d5a714d88b2 Nov 03 21:44:07.848703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:07.847041 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" event={"ID":"b1ab6e34-3152-4ece-a98e-4fc7f59ec674","Type":"ContainerStarted","Data":"167855d778ef2384cdf85aebe2a10a382d83d8c60b3cfc9f2f13a14ba135d1ed"} Nov 03 21:44:07.848703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:07.847110 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" event={"ID":"b1ab6e34-3152-4ece-a98e-4fc7f59ec674","Type":"ContainerStarted","Data":"33a46c4e74aad69d2dee37a706223e65f0f1815d412ea51d71616d5a714d88b2"} Nov 03 21:44:07.887845 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:07.887782 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" podStartSLOduration=1.887766544 podStartE2EDuration="1.887766544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:07.88668928 +0000 UTC m=+863.393917553" watchObservedRunningTime="2026-11-03 21:44:07.887766544 +0000 UTC m=+863.394994823" Nov 03 21:44:08.021191 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:08.021159 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp" Nov 03 21:44:08.066626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:08.064751 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-maas-api-xks-ca\") pod \"8ca4cd03-2b97-4cc8-9306-ea44a53e0668\" (UID: \"8ca4cd03-2b97-4cc8-9306-ea44a53e0668\") " Nov 03 21:44:08.066626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:08.064851 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-kube-api-access-h4v9r\" (UniqueName: \"kubernetes.io/projected/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-kube-api-access-h4v9r\") pod \"8ca4cd03-2b97-4cc8-9306-ea44a53e0668\" (UID: \"8ca4cd03-2b97-4cc8-9306-ea44a53e0668\") " Nov 03 21:44:08.066626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:08.064943 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-maas-api-openshift-ca\") pod \"8ca4cd03-2b97-4cc8-9306-ea44a53e0668\" (UID: \"8ca4cd03-2b97-4cc8-9306-ea44a53e0668\") " Nov 03 21:44:08.066626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:08.065940 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-maas-api-openshift-ca" pod "8ca4cd03-2b97-4cc8-9306-ea44a53e0668" (UID: "8ca4cd03-2b97-4cc8-9306-ea44a53e0668"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:08.069369 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:08.068612 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-maas-api-xks-ca" pod "8ca4cd03-2b97-4cc8-9306-ea44a53e0668" (UID: "8ca4cd03-2b97-4cc8-9306-ea44a53e0668"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:08.072752 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:08.072722 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-kube-api-access-h4v9r" pod "8ca4cd03-2b97-4cc8-9306-ea44a53e0668" (UID: "8ca4cd03-2b97-4cc8-9306-ea44a53e0668"). InnerVolumeSpecName "kube-api-access-h4v9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:08.166431 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:08.166396 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:08.166431 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:08.166426 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4v9r\" (UniqueName: \"kubernetes.io/projected/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-kube-api-access-h4v9r\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:08.166431 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:08.166437 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca4cd03-2b97-4cc8-9306-ea44a53e0668-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:08.857709 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:08.857670 5041 generic.go:356] "Generic (PLEG): container finished" podID="b1ab6e34-3152-4ece-a98e-4fc7f59ec674" containerID="167855d778ef2384cdf85aebe2a10a382d83d8c60b3cfc9f2f13a14ba135d1ed" exitCode=0 Nov 03 21:44:08.858147 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:08.857753 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" event={"ID":"b1ab6e34-3152-4ece-a98e-4fc7f59ec674","Type":"ContainerDied","Data":"167855d778ef2384cdf85aebe2a10a382d83d8c60b3cfc9f2f13a14ba135d1ed"} Nov 03 21:44:08.859596 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:08.859571 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp" event={"ID":"8ca4cd03-2b97-4cc8-9306-ea44a53e0668","Type":"ContainerDied","Data":"8a54994cb84a164b721993ddb353a6e583b6fc43721dcdd8745a88fc13eb4b02"} Nov 03 21:44:08.859734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:08.859613 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a54994cb84a164b721993ddb353a6e583b6fc43721dcdd8745a88fc13eb4b02" Nov 03 21:44:08.859734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:08.859583 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp" Nov 03 21:44:10.024399 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:10.024375 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" Nov 03 21:44:10.088014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:10.087985 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-kube-api-access-9zngx\" (UniqueName: \"kubernetes.io/projected/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-kube-api-access-9zngx\") pod \"b1ab6e34-3152-4ece-a98e-4fc7f59ec674\" (UID: \"b1ab6e34-3152-4ece-a98e-4fc7f59ec674\") " Nov 03 21:44:10.088205 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:10.088071 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-maas-api-xks-ca\") pod \"b1ab6e34-3152-4ece-a98e-4fc7f59ec674\" (UID: \"b1ab6e34-3152-4ece-a98e-4fc7f59ec674\") " Nov 03 21:44:10.088205 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:10.088101 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-maas-api-openshift-ca\") pod \"b1ab6e34-3152-4ece-a98e-4fc7f59ec674\" (UID: \"b1ab6e34-3152-4ece-a98e-4fc7f59ec674\") " Nov 03 21:44:10.088527 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:10.088488 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-maas-api-openshift-ca" pod "b1ab6e34-3152-4ece-a98e-4fc7f59ec674" (UID: "b1ab6e34-3152-4ece-a98e-4fc7f59ec674"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:10.090390 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:10.090363 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-maas-api-xks-ca" pod "b1ab6e34-3152-4ece-a98e-4fc7f59ec674" (UID: "b1ab6e34-3152-4ece-a98e-4fc7f59ec674"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:10.090796 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:10.090774 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-kube-api-access-9zngx" pod "b1ab6e34-3152-4ece-a98e-4fc7f59ec674" (UID: "b1ab6e34-3152-4ece-a98e-4fc7f59ec674"). InnerVolumeSpecName "kube-api-access-9zngx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:10.189806 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:10.189718 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zngx\" (UniqueName: \"kubernetes.io/projected/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-kube-api-access-9zngx\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:10.189806 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:10.189752 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:10.189806 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:10.189765 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1ab6e34-3152-4ece-a98e-4fc7f59ec674-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:10.779950 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:44:10.779911 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:44:10.876037 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:10.876006 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" Nov 03 21:44:10.876210 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:10.876006 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8" event={"ID":"b1ab6e34-3152-4ece-a98e-4fc7f59ec674","Type":"ContainerDied","Data":"33a46c4e74aad69d2dee37a706223e65f0f1815d412ea51d71616d5a714d88b2"} Nov 03 21:44:10.876210 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:10.876137 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33a46c4e74aad69d2dee37a706223e65f0f1815d412ea51d71616d5a714d88b2" Nov 03 21:44:16.708506 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.708377 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb"] Nov 03 21:44:16.710199 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.710155 5041 cpu_manager.go:414] "containerMap: removing container" podUID="8ca4cd03-2b97-4cc8-9306-ea44a53e0668" containerName="revoke-keys" Nov 03 21:44:16.710199 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.710200 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca4cd03-2b97-4cc8-9306-ea44a53e0668" containerName="revoke-keys" Nov 03 21:44:16.710427 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.710225 5041 cpu_manager.go:414] "containerMap: removing container" podUID="b1ab6e34-3152-4ece-a98e-4fc7f59ec674" containerName="revoke-keys" Nov 03 21:44:16.710427 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.710233 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ab6e34-3152-4ece-a98e-4fc7f59ec674" containerName="revoke-keys" Nov 03 21:44:16.710723 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.710705 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca4cd03-2b97-4cc8-9306-ea44a53e0668" containerName="revoke-keys" Nov 03 21:44:16.710815 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.710729 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ab6e34-3152-4ece-a98e-4fc7f59ec674" containerName="revoke-keys" Nov 03 21:44:16.722190 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.722164 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" Nov 03 21:44:16.732155 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.732128 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb"] Nov 03 21:44:16.741531 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.741503 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:44:16.741708 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.741556 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:44:16.809007 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.808963 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259c779d-03a3-46bb-9091-ba214b339c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d86cc3ff-m2nqb\" (UID: \"259c779d-03a3-46bb-9091-ba214b339c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" Nov 03 21:44:16.809212 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.809167 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259c779d-03a3-46bb-9091-ba214b339c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d86cc3ff-m2nqb\" (UID: \"259c779d-03a3-46bb-9091-ba214b339c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" Nov 03 21:44:16.809338 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.809316 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6vhg\" (UniqueName: \"kubernetes.io/projected/259c779d-03a3-46bb-9091-ba214b339c72-kube-api-access-t6vhg\") pod \"maas-api-revoke-sub-d86cc3ff-m2nqb\" (UID: \"259c779d-03a3-46bb-9091-ba214b339c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" Nov 03 21:44:16.911165 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.911130 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259c779d-03a3-46bb-9091-ba214b339c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d86cc3ff-m2nqb\" (UID: \"259c779d-03a3-46bb-9091-ba214b339c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" Nov 03 21:44:16.911351 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.911246 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6vhg\" (UniqueName: \"kubernetes.io/projected/259c779d-03a3-46bb-9091-ba214b339c72-kube-api-access-t6vhg\") pod \"maas-api-revoke-sub-d86cc3ff-m2nqb\" (UID: \"259c779d-03a3-46bb-9091-ba214b339c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" Nov 03 21:44:16.911351 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.911339 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259c779d-03a3-46bb-9091-ba214b339c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d86cc3ff-m2nqb\" (UID: \"259c779d-03a3-46bb-9091-ba214b339c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" Nov 03 21:44:16.911984 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.911953 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259c779d-03a3-46bb-9091-ba214b339c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d86cc3ff-m2nqb\" (UID: \"259c779d-03a3-46bb-9091-ba214b339c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" Nov 03 21:44:16.913814 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.913786 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259c779d-03a3-46bb-9091-ba214b339c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d86cc3ff-m2nqb\" (UID: \"259c779d-03a3-46bb-9091-ba214b339c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" Nov 03 21:44:16.931619 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:16.931587 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6vhg\" (UniqueName: \"kubernetes.io/projected/259c779d-03a3-46bb-9091-ba214b339c72-kube-api-access-t6vhg\") pod \"maas-api-revoke-sub-d86cc3ff-m2nqb\" (UID: \"259c779d-03a3-46bb-9091-ba214b339c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" Nov 03 21:44:17.037697 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.037581 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" Nov 03 21:44:17.345315 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:44:17.345277 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259c779d_03a3_46bb_9091_ba214b339c72.slice/crio-b4c285d78c14dd48656e80968b67c5a0745b302345a326b9a1595275eb01b3cc WatchSource:0}: Error finding container b4c285d78c14dd48656e80968b67c5a0745b302345a326b9a1595275eb01b3cc: Status 404 returned error can't find the container with id b4c285d78c14dd48656e80968b67c5a0745b302345a326b9a1595275eb01b3cc Nov 03 21:44:17.346076 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.346022 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb"] Nov 03 21:44:17.352880 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.352505 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g"] Nov 03 21:44:17.358166 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.358143 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" Nov 03 21:44:17.380736 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.379151 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g"] Nov 03 21:44:17.518580 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.518551 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f317155d-jn47g\" (UID: \"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" Nov 03 21:44:17.518770 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.518606 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f317155d-jn47g\" (UID: \"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" Nov 03 21:44:17.518840 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.518769 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k56sg\" (UniqueName: \"kubernetes.io/projected/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-kube-api-access-k56sg\") pod \"maas-api-revoke-sub-f317155d-jn47g\" (UID: \"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" Nov 03 21:44:17.620097 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.620067 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f317155d-jn47g\" (UID: \"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" Nov 03 21:44:17.620222 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.620123 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f317155d-jn47g\" (UID: \"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" Nov 03 21:44:17.620222 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.620174 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k56sg\" (UniqueName: \"kubernetes.io/projected/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-kube-api-access-k56sg\") pod \"maas-api-revoke-sub-f317155d-jn47g\" (UID: \"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" Nov 03 21:44:17.621036 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.621007 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f317155d-jn47g\" (UID: \"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" Nov 03 21:44:17.622613 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.622591 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f317155d-jn47g\" (UID: \"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" Nov 03 21:44:17.640366 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.640335 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k56sg\" (UniqueName: \"kubernetes.io/projected/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-kube-api-access-k56sg\") pod \"maas-api-revoke-sub-f317155d-jn47g\" (UID: \"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" Nov 03 21:44:17.677678 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.677632 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" Nov 03 21:44:17.944123 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.944029 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" event={"ID":"259c779d-03a3-46bb-9091-ba214b339c72","Type":"ContainerStarted","Data":"4d69c270a51d1d6aac46c4a43aa9b16732203cda47fb9f8b1ade0a48c22a7c81"} Nov 03 21:44:17.944123 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.944100 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" event={"ID":"259c779d-03a3-46bb-9091-ba214b339c72","Type":"ContainerStarted","Data":"b4c285d78c14dd48656e80968b67c5a0745b302345a326b9a1595275eb01b3cc"} Nov 03 21:44:17.978326 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.978294 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g"] Nov 03 21:44:17.978484 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:44:17.978358 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c3dbd5_96d7_4b66_a3a3_22f0de7b3c46.slice/crio-777f2fb6a8201639305b1128ae05429f9a0b349e9aff05ae36fb13b17f0f208c WatchSource:0}: Error finding container 777f2fb6a8201639305b1128ae05429f9a0b349e9aff05ae36fb13b17f0f208c: Status 404 returned error can't find the container with id 777f2fb6a8201639305b1128ae05429f9a0b349e9aff05ae36fb13b17f0f208c Nov 03 21:44:17.987365 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:17.987307 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" podStartSLOduration=1.987287893 podStartE2EDuration="1.987287893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:17.984147265 +0000 UTC m=+873.491375565" watchObservedRunningTime="2026-11-03 21:44:17.987287893 +0000 UTC m=+873.494516174" Nov 03 21:44:18.952790 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:18.952756 5041 generic.go:356] "Generic (PLEG): container finished" podID="259c779d-03a3-46bb-9091-ba214b339c72" containerID="4d69c270a51d1d6aac46c4a43aa9b16732203cda47fb9f8b1ade0a48c22a7c81" exitCode=0 Nov 03 21:44:18.953299 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:18.952837 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" event={"ID":"259c779d-03a3-46bb-9091-ba214b339c72","Type":"ContainerDied","Data":"4d69c270a51d1d6aac46c4a43aa9b16732203cda47fb9f8b1ade0a48c22a7c81"} Nov 03 21:44:18.954765 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:18.954739 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" event={"ID":"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46","Type":"ContainerStarted","Data":"be087dba4989585f7bcdc9e5dc308bfd954d34674ec67f0d0c2c1908528bd884"} Nov 03 21:44:18.954923 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:18.954782 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" event={"ID":"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46","Type":"ContainerStarted","Data":"777f2fb6a8201639305b1128ae05429f9a0b349e9aff05ae36fb13b17f0f208c"} Nov 03 21:44:19.034410 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:19.034361 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" podStartSLOduration=2.034344626 podStartE2EDuration="2.034344626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:19.033290295 +0000 UTC m=+874.540518600" watchObservedRunningTime="2026-11-03 21:44:19.034344626 +0000 UTC m=+874.541572906" Nov 03 21:44:19.553567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:19.553528 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x"] Nov 03 21:44:19.557495 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:19.557474 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" Nov 03 21:44:19.573678 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:19.573637 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x"] Nov 03 21:44:19.747434 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:19.747396 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30344d7-76ca-4674-a5d3-cf594b0ebdca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad385913-6fn9x\" (UID: \"d30344d7-76ca-4674-a5d3-cf594b0ebdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" Nov 03 21:44:19.747643 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:19.747458 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6drm\" (UniqueName: \"kubernetes.io/projected/d30344d7-76ca-4674-a5d3-cf594b0ebdca-kube-api-access-d6drm\") pod \"maas-api-revoke-sub-ad385913-6fn9x\" (UID: \"d30344d7-76ca-4674-a5d3-cf594b0ebdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" Nov 03 21:44:19.747643 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:19.747593 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30344d7-76ca-4674-a5d3-cf594b0ebdca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad385913-6fn9x\" (UID: \"d30344d7-76ca-4674-a5d3-cf594b0ebdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" Nov 03 21:44:19.849292 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:19.849203 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6drm\" (UniqueName: \"kubernetes.io/projected/d30344d7-76ca-4674-a5d3-cf594b0ebdca-kube-api-access-d6drm\") pod \"maas-api-revoke-sub-ad385913-6fn9x\" (UID: \"d30344d7-76ca-4674-a5d3-cf594b0ebdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" Nov 03 21:44:19.849292 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:19.849276 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30344d7-76ca-4674-a5d3-cf594b0ebdca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad385913-6fn9x\" (UID: \"d30344d7-76ca-4674-a5d3-cf594b0ebdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" Nov 03 21:44:19.849528 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:19.849388 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30344d7-76ca-4674-a5d3-cf594b0ebdca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad385913-6fn9x\" (UID: \"d30344d7-76ca-4674-a5d3-cf594b0ebdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" Nov 03 21:44:19.850046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:19.850025 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30344d7-76ca-4674-a5d3-cf594b0ebdca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad385913-6fn9x\" (UID: \"d30344d7-76ca-4674-a5d3-cf594b0ebdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" Nov 03 21:44:19.851887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:19.851865 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30344d7-76ca-4674-a5d3-cf594b0ebdca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad385913-6fn9x\" (UID: \"d30344d7-76ca-4674-a5d3-cf594b0ebdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" Nov 03 21:44:19.869394 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:19.869369 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6drm\" (UniqueName: \"kubernetes.io/projected/d30344d7-76ca-4674-a5d3-cf594b0ebdca-kube-api-access-d6drm\") pod \"maas-api-revoke-sub-ad385913-6fn9x\" (UID: \"d30344d7-76ca-4674-a5d3-cf594b0ebdca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" Nov 03 21:44:19.873287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:19.873261 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" Nov 03 21:44:19.966623 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:19.966584 5041 generic.go:356] "Generic (PLEG): container finished" podID="52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46" containerID="be087dba4989585f7bcdc9e5dc308bfd954d34674ec67f0d0c2c1908528bd884" exitCode=0 Nov 03 21:44:19.967071 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:19.966701 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" event={"ID":"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46","Type":"ContainerDied","Data":"be087dba4989585f7bcdc9e5dc308bfd954d34674ec67f0d0c2c1908528bd884"} Nov 03 21:44:20.178094 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:20.178066 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x"] Nov 03 21:44:20.179373 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:44:20.179340 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30344d7_76ca_4674_a5d3_cf594b0ebdca.slice/crio-9f1d10845adb439fffbcf151fc49d99d2c9016820d0db5d88ef23b0bf85c28d1 WatchSource:0}: Error finding container 9f1d10845adb439fffbcf151fc49d99d2c9016820d0db5d88ef23b0bf85c28d1: Status 404 returned error can't find the container with id 9f1d10845adb439fffbcf151fc49d99d2c9016820d0db5d88ef23b0bf85c28d1 Nov 03 21:44:20.201803 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:20.201779 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" Nov 03 21:44:20.360707 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:20.360647 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/259c779d-03a3-46bb-9091-ba214b339c72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259c779d-03a3-46bb-9091-ba214b339c72-maas-api-openshift-ca\") pod \"259c779d-03a3-46bb-9091-ba214b339c72\" (UID: \"259c779d-03a3-46bb-9091-ba214b339c72\") " Nov 03 21:44:20.360894 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:20.360835 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/259c779d-03a3-46bb-9091-ba214b339c72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259c779d-03a3-46bb-9091-ba214b339c72-maas-api-xks-ca\") pod \"259c779d-03a3-46bb-9091-ba214b339c72\" (UID: \"259c779d-03a3-46bb-9091-ba214b339c72\") " Nov 03 21:44:20.361002 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:20.360989 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/259c779d-03a3-46bb-9091-ba214b339c72-kube-api-access-t6vhg\" (UniqueName: \"kubernetes.io/projected/259c779d-03a3-46bb-9091-ba214b339c72-kube-api-access-t6vhg\") pod \"259c779d-03a3-46bb-9091-ba214b339c72\" (UID: \"259c779d-03a3-46bb-9091-ba214b339c72\") " Nov 03 21:44:20.361070 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:20.360994 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259c779d-03a3-46bb-9091-ba214b339c72-maas-api-openshift-ca" pod "259c779d-03a3-46bb-9091-ba214b339c72" (UID: "259c779d-03a3-46bb-9091-ba214b339c72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:20.361320 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:20.361302 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259c779d-03a3-46bb-9091-ba214b339c72-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:20.362872 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:20.362848 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/259c779d-03a3-46bb-9091-ba214b339c72-maas-api-xks-ca" pod "259c779d-03a3-46bb-9091-ba214b339c72" (UID: "259c779d-03a3-46bb-9091-ba214b339c72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:20.363020 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:20.362993 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259c779d-03a3-46bb-9091-ba214b339c72-kube-api-access-t6vhg" pod "259c779d-03a3-46bb-9091-ba214b339c72" (UID: "259c779d-03a3-46bb-9091-ba214b339c72"). InnerVolumeSpecName "kube-api-access-t6vhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:20.462090 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:20.462053 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259c779d-03a3-46bb-9091-ba214b339c72-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:20.462090 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:20.462083 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6vhg\" (UniqueName: \"kubernetes.io/projected/259c779d-03a3-46bb-9091-ba214b339c72-kube-api-access-t6vhg\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:20.978447 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:20.978415 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" Nov 03 21:44:20.978991 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:20.978420 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb" event={"ID":"259c779d-03a3-46bb-9091-ba214b339c72","Type":"ContainerDied","Data":"b4c285d78c14dd48656e80968b67c5a0745b302345a326b9a1595275eb01b3cc"} Nov 03 21:44:20.978991 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:20.978563 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4c285d78c14dd48656e80968b67c5a0745b302345a326b9a1595275eb01b3cc" Nov 03 21:44:20.981525 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:20.981492 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" event={"ID":"d30344d7-76ca-4674-a5d3-cf594b0ebdca","Type":"ContainerStarted","Data":"1ff9fab001a1700e83f684e8e9c7b90eaf63e1319532aeb5f6bb412fe9a8f823"} Nov 03 21:44:20.981715 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:20.981550 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" event={"ID":"d30344d7-76ca-4674-a5d3-cf594b0ebdca","Type":"ContainerStarted","Data":"9f1d10845adb439fffbcf151fc49d99d2c9016820d0db5d88ef23b0bf85c28d1"} Nov 03 21:44:21.029984 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.029930 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" podStartSLOduration=2.029915525 podStartE2EDuration="2.029915525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:21.020736787 +0000 UTC m=+876.527965063" watchObservedRunningTime="2026-11-03 21:44:21.029915525 +0000 UTC m=+876.537143804" Nov 03 21:44:21.153679 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.153627 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" Nov 03 21:44:21.169445 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.169413 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-maas-api-xks-ca\") pod \"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46\" (UID: \"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46\") " Nov 03 21:44:21.169617 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.169558 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-maas-api-openshift-ca\") pod \"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46\" (UID: \"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46\") " Nov 03 21:44:21.169617 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.169581 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-kube-api-access-k56sg\" (UniqueName: \"kubernetes.io/projected/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-kube-api-access-k56sg\") pod \"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46\" (UID: \"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46\") " Nov 03 21:44:21.169974 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.169949 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-maas-api-openshift-ca" pod "52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46" (UID: "52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:21.170150 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.170135 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:21.171703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.171649 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-maas-api-xks-ca" pod "52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46" (UID: "52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:21.171811 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.171739 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-kube-api-access-k56sg" pod "52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46" (UID: "52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46"). InnerVolumeSpecName "kube-api-access-k56sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:21.200343 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.200256 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68"] Nov 03 21:44:21.201343 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.201315 5041 cpu_manager.go:414] "containerMap: removing container" podUID="259c779d-03a3-46bb-9091-ba214b339c72" containerName="revoke-keys" Nov 03 21:44:21.201471 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.201342 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="259c779d-03a3-46bb-9091-ba214b339c72" containerName="revoke-keys" Nov 03 21:44:21.201471 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.201363 5041 cpu_manager.go:414] "containerMap: removing container" podUID="52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46" containerName="revoke-keys" Nov 03 21:44:21.201471 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.201371 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46" containerName="revoke-keys" Nov 03 21:44:21.201641 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.201597 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="259c779d-03a3-46bb-9091-ba214b339c72" containerName="revoke-keys" Nov 03 21:44:21.201641 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.201616 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46" containerName="revoke-keys" Nov 03 21:44:21.205884 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.205860 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68" Nov 03 21:44:21.229347 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.224102 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68"] Nov 03 21:44:21.270877 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.270839 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb5f8ada-8543-4578-a8d9-a05d590b9982-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47c651af-j8f68\" (UID: \"cb5f8ada-8543-4578-a8d9-a05d590b9982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68" Nov 03 21:44:21.271060 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.271037 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb5f8ada-8543-4578-a8d9-a05d590b9982-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47c651af-j8f68\" (UID: \"cb5f8ada-8543-4578-a8d9-a05d590b9982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68" Nov 03 21:44:21.271165 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.271144 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2rm8\" (UniqueName: \"kubernetes.io/projected/cb5f8ada-8543-4578-a8d9-a05d590b9982-kube-api-access-n2rm8\") pod \"maas-api-revoke-sub-47c651af-j8f68\" (UID: \"cb5f8ada-8543-4578-a8d9-a05d590b9982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68" Nov 03 21:44:21.271239 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.271225 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k56sg\" (UniqueName: \"kubernetes.io/projected/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-kube-api-access-k56sg\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:21.271275 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.271246 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:21.372213 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.372175 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb5f8ada-8543-4578-a8d9-a05d590b9982-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47c651af-j8f68\" (UID: \"cb5f8ada-8543-4578-a8d9-a05d590b9982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68" Nov 03 21:44:21.372357 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.372259 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2rm8\" (UniqueName: \"kubernetes.io/projected/cb5f8ada-8543-4578-a8d9-a05d590b9982-kube-api-access-n2rm8\") pod \"maas-api-revoke-sub-47c651af-j8f68\" (UID: \"cb5f8ada-8543-4578-a8d9-a05d590b9982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68" Nov 03 21:44:21.372357 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.372287 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb5f8ada-8543-4578-a8d9-a05d590b9982-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47c651af-j8f68\" (UID: \"cb5f8ada-8543-4578-a8d9-a05d590b9982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68" Nov 03 21:44:21.373021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.372999 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb5f8ada-8543-4578-a8d9-a05d590b9982-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47c651af-j8f68\" (UID: \"cb5f8ada-8543-4578-a8d9-a05d590b9982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68" Nov 03 21:44:21.374735 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.374713 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb5f8ada-8543-4578-a8d9-a05d590b9982-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47c651af-j8f68\" (UID: \"cb5f8ada-8543-4578-a8d9-a05d590b9982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68" Nov 03 21:44:21.392935 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.392905 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2rm8\" (UniqueName: \"kubernetes.io/projected/cb5f8ada-8543-4578-a8d9-a05d590b9982-kube-api-access-n2rm8\") pod \"maas-api-revoke-sub-47c651af-j8f68\" (UID: \"cb5f8ada-8543-4578-a8d9-a05d590b9982\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68" Nov 03 21:44:21.530700 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.530600 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68" Nov 03 21:44:21.832192 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.832151 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68"] Nov 03 21:44:21.833264 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:44:21.833239 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5f8ada_8543_4578_a8d9_a05d590b9982.slice/crio-164cc328ecf2bd33a8046b400e454f612cb9ca3d8ebe72725fcf6b0bc8ed55dc WatchSource:0}: Error finding container 164cc328ecf2bd33a8046b400e454f612cb9ca3d8ebe72725fcf6b0bc8ed55dc: Status 404 returned error can't find the container with id 164cc328ecf2bd33a8046b400e454f612cb9ca3d8ebe72725fcf6b0bc8ed55dc Nov 03 21:44:21.992098 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.992067 5041 generic.go:356] "Generic (PLEG): container finished" podID="d30344d7-76ca-4674-a5d3-cf594b0ebdca" containerID="1ff9fab001a1700e83f684e8e9c7b90eaf63e1319532aeb5f6bb412fe9a8f823" exitCode=0 Nov 03 21:44:21.992580 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.992150 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" event={"ID":"d30344d7-76ca-4674-a5d3-cf594b0ebdca","Type":"ContainerDied","Data":"1ff9fab001a1700e83f684e8e9c7b90eaf63e1319532aeb5f6bb412fe9a8f823"} Nov 03 21:44:21.993729 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.993695 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" event={"ID":"52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46","Type":"ContainerDied","Data":"777f2fb6a8201639305b1128ae05429f9a0b349e9aff05ae36fb13b17f0f208c"} Nov 03 21:44:21.993869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.993748 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="777f2fb6a8201639305b1128ae05429f9a0b349e9aff05ae36fb13b17f0f208c" Nov 03 21:44:21.993869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.993707 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g" Nov 03 21:44:21.995132 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:21.995112 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68" event={"ID":"cb5f8ada-8543-4578-a8d9-a05d590b9982","Type":"ContainerStarted","Data":"164cc328ecf2bd33a8046b400e454f612cb9ca3d8ebe72725fcf6b0bc8ed55dc"} Nov 03 21:44:22.596538 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:22.596458 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl"] Nov 03 21:44:22.600649 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:22.600629 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" Nov 03 21:44:22.636721 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:22.636689 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl"] Nov 03 21:44:22.700272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:22.700238 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jz8q\" (UniqueName: \"kubernetes.io/projected/479af13b-6f89-4a24-9fd5-619888b129cc-kube-api-access-8jz8q\") pod \"maas-api-revoke-sub-b4168ce3-2r5gl\" (UID: \"479af13b-6f89-4a24-9fd5-619888b129cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" Nov 03 21:44:22.700272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:22.700278 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479af13b-6f89-4a24-9fd5-619888b129cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4168ce3-2r5gl\" (UID: \"479af13b-6f89-4a24-9fd5-619888b129cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" Nov 03 21:44:22.700489 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:22.700298 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479af13b-6f89-4a24-9fd5-619888b129cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4168ce3-2r5gl\" (UID: \"479af13b-6f89-4a24-9fd5-619888b129cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" Nov 03 21:44:22.802222 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:22.802181 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jz8q\" (UniqueName: \"kubernetes.io/projected/479af13b-6f89-4a24-9fd5-619888b129cc-kube-api-access-8jz8q\") pod \"maas-api-revoke-sub-b4168ce3-2r5gl\" (UID: \"479af13b-6f89-4a24-9fd5-619888b129cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" Nov 03 21:44:22.802222 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:22.802223 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479af13b-6f89-4a24-9fd5-619888b129cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4168ce3-2r5gl\" (UID: \"479af13b-6f89-4a24-9fd5-619888b129cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" Nov 03 21:44:22.802432 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:22.802243 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479af13b-6f89-4a24-9fd5-619888b129cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4168ce3-2r5gl\" (UID: \"479af13b-6f89-4a24-9fd5-619888b129cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" Nov 03 21:44:22.802909 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:22.802889 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479af13b-6f89-4a24-9fd5-619888b129cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4168ce3-2r5gl\" (UID: \"479af13b-6f89-4a24-9fd5-619888b129cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" Nov 03 21:44:22.804715 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:22.804692 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479af13b-6f89-4a24-9fd5-619888b129cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4168ce3-2r5gl\" (UID: \"479af13b-6f89-4a24-9fd5-619888b129cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" Nov 03 21:44:22.822565 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:22.822540 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jz8q\" (UniqueName: \"kubernetes.io/projected/479af13b-6f89-4a24-9fd5-619888b129cc-kube-api-access-8jz8q\") pod \"maas-api-revoke-sub-b4168ce3-2r5gl\" (UID: \"479af13b-6f89-4a24-9fd5-619888b129cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" Nov 03 21:44:22.940175 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:22.940077 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" Nov 03 21:44:23.020557 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.019714 5041 generic.go:356] "Generic (PLEG): container finished" podID="cb5f8ada-8543-4578-a8d9-a05d590b9982" containerID="340ec8c87542889b8c191685c4b98a8e4805bafffbe0588f26b1e19ab01c5c38" exitCode=0 Nov 03 21:44:23.020557 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.020466 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68" event={"ID":"cb5f8ada-8543-4578-a8d9-a05d590b9982","Type":"ContainerDied","Data":"340ec8c87542889b8c191685c4b98a8e4805bafffbe0588f26b1e19ab01c5c38"} Nov 03 21:44:23.226016 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.225985 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" Nov 03 21:44:23.245388 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:44:23.245350 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479af13b_6f89_4a24_9fd5_619888b129cc.slice/crio-641a7eec899f74dd125db312e293d7112e58db4bbf16c5fb829b99511cdea156 WatchSource:0}: Error finding container 641a7eec899f74dd125db312e293d7112e58db4bbf16c5fb829b99511cdea156: Status 404 returned error can't find the container with id 641a7eec899f74dd125db312e293d7112e58db4bbf16c5fb829b99511cdea156 Nov 03 21:44:23.245545 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.245489 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl"] Nov 03 21:44:23.308203 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.308172 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d30344d7-76ca-4674-a5d3-cf594b0ebdca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30344d7-76ca-4674-a5d3-cf594b0ebdca-maas-api-xks-ca\") pod \"d30344d7-76ca-4674-a5d3-cf594b0ebdca\" (UID: \"d30344d7-76ca-4674-a5d3-cf594b0ebdca\") " Nov 03 21:44:23.308600 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.308569 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d30344d7-76ca-4674-a5d3-cf594b0ebdca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30344d7-76ca-4674-a5d3-cf594b0ebdca-maas-api-openshift-ca\") pod \"d30344d7-76ca-4674-a5d3-cf594b0ebdca\" (UID: \"d30344d7-76ca-4674-a5d3-cf594b0ebdca\") " Nov 03 21:44:23.308738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.308636 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30344d7-76ca-4674-a5d3-cf594b0ebdca-kube-api-access-d6drm\" (UniqueName: \"kubernetes.io/projected/d30344d7-76ca-4674-a5d3-cf594b0ebdca-kube-api-access-d6drm\") pod \"d30344d7-76ca-4674-a5d3-cf594b0ebdca\" (UID: \"d30344d7-76ca-4674-a5d3-cf594b0ebdca\") " Nov 03 21:44:23.309224 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.309171 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30344d7-76ca-4674-a5d3-cf594b0ebdca-maas-api-openshift-ca" pod "d30344d7-76ca-4674-a5d3-cf594b0ebdca" (UID: "d30344d7-76ca-4674-a5d3-cf594b0ebdca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:23.310893 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.310867 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d30344d7-76ca-4674-a5d3-cf594b0ebdca-maas-api-xks-ca" pod "d30344d7-76ca-4674-a5d3-cf594b0ebdca" (UID: "d30344d7-76ca-4674-a5d3-cf594b0ebdca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:23.311980 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.311955 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30344d7-76ca-4674-a5d3-cf594b0ebdca-kube-api-access-d6drm" pod "d30344d7-76ca-4674-a5d3-cf594b0ebdca" (UID: "d30344d7-76ca-4674-a5d3-cf594b0ebdca"). InnerVolumeSpecName "kube-api-access-d6drm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:23.410541 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.410515 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30344d7-76ca-4674-a5d3-cf594b0ebdca-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:23.410541 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.410539 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30344d7-76ca-4674-a5d3-cf594b0ebdca-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:23.410742 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.410550 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6drm\" (UniqueName: \"kubernetes.io/projected/d30344d7-76ca-4674-a5d3-cf594b0ebdca-kube-api-access-d6drm\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:23.703800 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.703769 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst"] Nov 03 21:44:23.704450 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.704433 5041 cpu_manager.go:414] "containerMap: removing container" podUID="d30344d7-76ca-4674-a5d3-cf594b0ebdca" containerName="revoke-keys" Nov 03 21:44:23.704517 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.704453 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30344d7-76ca-4674-a5d3-cf594b0ebdca" containerName="revoke-keys" Nov 03 21:44:23.704603 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.704593 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30344d7-76ca-4674-a5d3-cf594b0ebdca" containerName="revoke-keys" Nov 03 21:44:23.710606 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.710578 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" Nov 03 21:44:23.727910 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.727878 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst"] Nov 03 21:44:23.815678 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.815632 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2afffa16-69bst\" (UID: \"58e4dad7-bdde-4909-af0d-4f8eb8f145e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" Nov 03 21:44:23.815884 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.815856 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2afffa16-69bst\" (UID: \"58e4dad7-bdde-4909-af0d-4f8eb8f145e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" Nov 03 21:44:23.815931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.815901 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwsnc\" (UniqueName: \"kubernetes.io/projected/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-kube-api-access-cwsnc\") pod \"maas-api-revoke-sub-2afffa16-69bst\" (UID: \"58e4dad7-bdde-4909-af0d-4f8eb8f145e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" Nov 03 21:44:23.916972 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.916889 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2afffa16-69bst\" (UID: \"58e4dad7-bdde-4909-af0d-4f8eb8f145e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" Nov 03 21:44:23.916972 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.916927 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwsnc\" (UniqueName: \"kubernetes.io/projected/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-kube-api-access-cwsnc\") pod \"maas-api-revoke-sub-2afffa16-69bst\" (UID: \"58e4dad7-bdde-4909-af0d-4f8eb8f145e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" Nov 03 21:44:23.917196 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.917077 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2afffa16-69bst\" (UID: \"58e4dad7-bdde-4909-af0d-4f8eb8f145e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" Nov 03 21:44:23.917780 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.917757 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2afffa16-69bst\" (UID: \"58e4dad7-bdde-4909-af0d-4f8eb8f145e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" Nov 03 21:44:23.919329 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.919307 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2afffa16-69bst\" (UID: \"58e4dad7-bdde-4909-af0d-4f8eb8f145e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" Nov 03 21:44:23.936994 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:23.936971 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwsnc\" (UniqueName: \"kubernetes.io/projected/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-kube-api-access-cwsnc\") pod \"maas-api-revoke-sub-2afffa16-69bst\" (UID: \"58e4dad7-bdde-4909-af0d-4f8eb8f145e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" Nov 03 21:44:24.030503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.030473 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" Nov 03 21:44:24.030991 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.030507 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x" event={"ID":"d30344d7-76ca-4674-a5d3-cf594b0ebdca","Type":"ContainerDied","Data":"9f1d10845adb439fffbcf151fc49d99d2c9016820d0db5d88ef23b0bf85c28d1"} Nov 03 21:44:24.030991 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.030565 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f1d10845adb439fffbcf151fc49d99d2c9016820d0db5d88ef23b0bf85c28d1" Nov 03 21:44:24.032741 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.032463 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" event={"ID":"479af13b-6f89-4a24-9fd5-619888b129cc","Type":"ContainerStarted","Data":"47218e364eb304fc56fe76e7eaebd86a491918dda9ff89764bf85bb2e4c17024"} Nov 03 21:44:24.033031 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.032999 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" event={"ID":"479af13b-6f89-4a24-9fd5-619888b129cc","Type":"ContainerStarted","Data":"641a7eec899f74dd125db312e293d7112e58db4bbf16c5fb829b99511cdea156"} Nov 03 21:44:24.034184 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.034162 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" Nov 03 21:44:24.075731 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.075447 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" podStartSLOduration=2.075427246 podStartE2EDuration="2.075427246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:24.074217198 +0000 UTC m=+879.581445506" watchObservedRunningTime="2026-11-03 21:44:24.075427246 +0000 UTC m=+879.582655574" Nov 03 21:44:24.317899 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.317872 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68" Nov 03 21:44:24.347282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.347254 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst"] Nov 03 21:44:24.349505 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:44:24.349476 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e4dad7_bdde_4909_af0d_4f8eb8f145e5.slice/crio-be27a979f1d89bc763ed110eff578b73761b7a1471e70f0839db8f1f6f471039 WatchSource:0}: Error finding container be27a979f1d89bc763ed110eff578b73761b7a1471e70f0839db8f1f6f471039: Status 404 returned error can't find the container with id be27a979f1d89bc763ed110eff578b73761b7a1471e70f0839db8f1f6f471039 Nov 03 21:44:24.423264 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.423169 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb5f8ada-8543-4578-a8d9-a05d590b9982-kube-api-access-n2rm8\" (UniqueName: \"kubernetes.io/projected/cb5f8ada-8543-4578-a8d9-a05d590b9982-kube-api-access-n2rm8\") pod \"cb5f8ada-8543-4578-a8d9-a05d590b9982\" (UID: \"cb5f8ada-8543-4578-a8d9-a05d590b9982\") " Nov 03 21:44:24.423438 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.423270 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb5f8ada-8543-4578-a8d9-a05d590b9982-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb5f8ada-8543-4578-a8d9-a05d590b9982-maas-api-openshift-ca\") pod \"cb5f8ada-8543-4578-a8d9-a05d590b9982\" (UID: \"cb5f8ada-8543-4578-a8d9-a05d590b9982\") " Nov 03 21:44:24.423496 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.423467 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb5f8ada-8543-4578-a8d9-a05d590b9982-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb5f8ada-8543-4578-a8d9-a05d590b9982-maas-api-xks-ca\") pod \"cb5f8ada-8543-4578-a8d9-a05d590b9982\" (UID: \"cb5f8ada-8543-4578-a8d9-a05d590b9982\") " Nov 03 21:44:24.423645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.423621 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5f8ada-8543-4578-a8d9-a05d590b9982-maas-api-openshift-ca" pod "cb5f8ada-8543-4578-a8d9-a05d590b9982" (UID: "cb5f8ada-8543-4578-a8d9-a05d590b9982"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:24.423916 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.423886 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb5f8ada-8543-4578-a8d9-a05d590b9982-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:24.425868 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.425845 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5f8ada-8543-4578-a8d9-a05d590b9982-kube-api-access-n2rm8" pod "cb5f8ada-8543-4578-a8d9-a05d590b9982" (UID: "cb5f8ada-8543-4578-a8d9-a05d590b9982"). InnerVolumeSpecName "kube-api-access-n2rm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:24.425936 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.425907 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb5f8ada-8543-4578-a8d9-a05d590b9982-maas-api-xks-ca" pod "cb5f8ada-8543-4578-a8d9-a05d590b9982" (UID: "cb5f8ada-8543-4578-a8d9-a05d590b9982"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:24.517578 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.517547 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6"] Nov 03 21:44:24.518328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.518304 5041 cpu_manager.go:414] "containerMap: removing container" podUID="cb5f8ada-8543-4578-a8d9-a05d590b9982" containerName="revoke-keys" Nov 03 21:44:24.518328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.518327 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb5f8ada-8543-4578-a8d9-a05d590b9982" containerName="revoke-keys" Nov 03 21:44:24.518500 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.518477 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb5f8ada-8543-4578-a8d9-a05d590b9982" containerName="revoke-keys" Nov 03 21:44:24.522074 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.522047 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" Nov 03 21:44:24.525580 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.525555 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb5f8ada-8543-4578-a8d9-a05d590b9982-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:24.525716 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.525585 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2rm8\" (UniqueName: \"kubernetes.io/projected/cb5f8ada-8543-4578-a8d9-a05d590b9982-kube-api-access-n2rm8\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:24.549547 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.549515 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6"] Nov 03 21:44:24.626947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.626923 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6gkl\" (UniqueName: \"kubernetes.io/projected/0a8d742e-582e-461c-93ce-26b511bfed39-kube-api-access-j6gkl\") pod \"maas-api-revoke-sub-4c087459-ds7d6\" (UID: \"0a8d742e-582e-461c-93ce-26b511bfed39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" Nov 03 21:44:24.627144 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.627118 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8d742e-582e-461c-93ce-26b511bfed39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c087459-ds7d6\" (UID: \"0a8d742e-582e-461c-93ce-26b511bfed39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" Nov 03 21:44:24.627226 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.627211 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8d742e-582e-461c-93ce-26b511bfed39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c087459-ds7d6\" (UID: \"0a8d742e-582e-461c-93ce-26b511bfed39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" Nov 03 21:44:24.728543 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.728503 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8d742e-582e-461c-93ce-26b511bfed39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c087459-ds7d6\" (UID: \"0a8d742e-582e-461c-93ce-26b511bfed39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" Nov 03 21:44:24.728871 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.728840 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8d742e-582e-461c-93ce-26b511bfed39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c087459-ds7d6\" (UID: \"0a8d742e-582e-461c-93ce-26b511bfed39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" Nov 03 21:44:24.729007 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.728994 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6gkl\" (UniqueName: \"kubernetes.io/projected/0a8d742e-582e-461c-93ce-26b511bfed39-kube-api-access-j6gkl\") pod \"maas-api-revoke-sub-4c087459-ds7d6\" (UID: \"0a8d742e-582e-461c-93ce-26b511bfed39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" Nov 03 21:44:24.729695 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.729643 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8d742e-582e-461c-93ce-26b511bfed39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c087459-ds7d6\" (UID: \"0a8d742e-582e-461c-93ce-26b511bfed39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" Nov 03 21:44:24.731641 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.731619 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8d742e-582e-461c-93ce-26b511bfed39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c087459-ds7d6\" (UID: \"0a8d742e-582e-461c-93ce-26b511bfed39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" Nov 03 21:44:24.749178 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.749154 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6gkl\" (UniqueName: \"kubernetes.io/projected/0a8d742e-582e-461c-93ce-26b511bfed39-kube-api-access-j6gkl\") pod \"maas-api-revoke-sub-4c087459-ds7d6\" (UID: \"0a8d742e-582e-461c-93ce-26b511bfed39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" Nov 03 21:44:24.842367 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:24.842331 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" Nov 03 21:44:25.057784 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:25.057691 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" event={"ID":"58e4dad7-bdde-4909-af0d-4f8eb8f145e5","Type":"ContainerStarted","Data":"bfad6edb99945282362a118f4b219c12587322df2b077bb8ab82436574533ce5"} Nov 03 21:44:25.058240 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:25.057791 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" event={"ID":"58e4dad7-bdde-4909-af0d-4f8eb8f145e5","Type":"ContainerStarted","Data":"be27a979f1d89bc763ed110eff578b73761b7a1471e70f0839db8f1f6f471039"} Nov 03 21:44:25.060745 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:25.060719 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68" Nov 03 21:44:25.060874 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:25.060749 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68" event={"ID":"cb5f8ada-8543-4578-a8d9-a05d590b9982","Type":"ContainerDied","Data":"164cc328ecf2bd33a8046b400e454f612cb9ca3d8ebe72725fcf6b0bc8ed55dc"} Nov 03 21:44:25.060874 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:25.060798 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="164cc328ecf2bd33a8046b400e454f612cb9ca3d8ebe72725fcf6b0bc8ed55dc" Nov 03 21:44:25.062699 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:25.062657 5041 generic.go:356] "Generic (PLEG): container finished" podID="479af13b-6f89-4a24-9fd5-619888b129cc" containerID="47218e364eb304fc56fe76e7eaebd86a491918dda9ff89764bf85bb2e4c17024" exitCode=0 Nov 03 21:44:25.062829 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:25.062754 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" event={"ID":"479af13b-6f89-4a24-9fd5-619888b129cc","Type":"ContainerDied","Data":"47218e364eb304fc56fe76e7eaebd86a491918dda9ff89764bf85bb2e4c17024"} Nov 03 21:44:25.099266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:25.099219 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" podStartSLOduration=2.099205842 podStartE2EDuration="2.099205842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:25.096517998 +0000 UTC m=+880.603746298" watchObservedRunningTime="2026-11-03 21:44:25.099205842 +0000 UTC m=+880.606434137" Nov 03 21:44:25.162630 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:25.162601 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6"] Nov 03 21:44:26.072075 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:26.071988 5041 generic.go:356] "Generic (PLEG): container finished" podID="58e4dad7-bdde-4909-af0d-4f8eb8f145e5" containerID="bfad6edb99945282362a118f4b219c12587322df2b077bb8ab82436574533ce5" exitCode=0 Nov 03 21:44:26.072488 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:26.072078 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" event={"ID":"58e4dad7-bdde-4909-af0d-4f8eb8f145e5","Type":"ContainerDied","Data":"bfad6edb99945282362a118f4b219c12587322df2b077bb8ab82436574533ce5"} Nov 03 21:44:26.074389 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:26.074291 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" event={"ID":"0a8d742e-582e-461c-93ce-26b511bfed39","Type":"ContainerStarted","Data":"78a7f0e04c5d1c8b59c50b80fe599515aae7e2f4d8947800eb956f3930318ad9"} Nov 03 21:44:26.074389 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:26.074338 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" event={"ID":"0a8d742e-582e-461c-93ce-26b511bfed39","Type":"ContainerStarted","Data":"4af9f923e47e4177227d12cf8bda5399305add03d9c0ec1b8f262e7df2f75cf3"} Nov 03 21:44:26.162724 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:26.159804 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" podStartSLOduration=2.159784136 podStartE2EDuration="2.159784136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:26.1569122 +0000 UTC m=+881.664140494" watchObservedRunningTime="2026-11-03 21:44:26.159784136 +0000 UTC m=+881.667012418" Nov 03 21:44:26.324043 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:26.323955 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" Nov 03 21:44:26.364334 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:26.364294 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479af13b-6f89-4a24-9fd5-619888b129cc-kube-api-access-8jz8q\" (UniqueName: \"kubernetes.io/projected/479af13b-6f89-4a24-9fd5-619888b129cc-kube-api-access-8jz8q\") pod \"479af13b-6f89-4a24-9fd5-619888b129cc\" (UID: \"479af13b-6f89-4a24-9fd5-619888b129cc\") " Nov 03 21:44:26.365143 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:26.365116 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/479af13b-6f89-4a24-9fd5-619888b129cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479af13b-6f89-4a24-9fd5-619888b129cc-maas-api-xks-ca\") pod \"479af13b-6f89-4a24-9fd5-619888b129cc\" (UID: \"479af13b-6f89-4a24-9fd5-619888b129cc\") " Nov 03 21:44:26.365425 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:26.365403 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479af13b-6f89-4a24-9fd5-619888b129cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479af13b-6f89-4a24-9fd5-619888b129cc-maas-api-openshift-ca\") pod \"479af13b-6f89-4a24-9fd5-619888b129cc\" (UID: \"479af13b-6f89-4a24-9fd5-619888b129cc\") " Nov 03 21:44:26.366571 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:26.366549 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479af13b-6f89-4a24-9fd5-619888b129cc-maas-api-openshift-ca" pod "479af13b-6f89-4a24-9fd5-619888b129cc" (UID: "479af13b-6f89-4a24-9fd5-619888b129cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:26.393725 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:26.389224 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/479af13b-6f89-4a24-9fd5-619888b129cc-maas-api-xks-ca" pod "479af13b-6f89-4a24-9fd5-619888b129cc" (UID: "479af13b-6f89-4a24-9fd5-619888b129cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:26.418990 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:26.418940 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479af13b-6f89-4a24-9fd5-619888b129cc-kube-api-access-8jz8q" pod "479af13b-6f89-4a24-9fd5-619888b129cc" (UID: "479af13b-6f89-4a24-9fd5-619888b129cc"). InnerVolumeSpecName "kube-api-access-8jz8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:26.471845 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:26.471805 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479af13b-6f89-4a24-9fd5-619888b129cc-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:26.472056 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:26.472035 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479af13b-6f89-4a24-9fd5-619888b129cc-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:26.472176 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:26.472161 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jz8q\" (UniqueName: \"kubernetes.io/projected/479af13b-6f89-4a24-9fd5-619888b129cc-kube-api-access-8jz8q\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:27.134431 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:27.134388 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" event={"ID":"479af13b-6f89-4a24-9fd5-619888b129cc","Type":"ContainerDied","Data":"641a7eec899f74dd125db312e293d7112e58db4bbf16c5fb829b99511cdea156"} Nov 03 21:44:27.135075 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:27.135053 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="641a7eec899f74dd125db312e293d7112e58db4bbf16c5fb829b99511cdea156" Nov 03 21:44:27.135290 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:27.135274 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl" Nov 03 21:44:27.166201 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:27.166167 5041 generic.go:356] "Generic (PLEG): container finished" podID="0a8d742e-582e-461c-93ce-26b511bfed39" containerID="78a7f0e04c5d1c8b59c50b80fe599515aae7e2f4d8947800eb956f3930318ad9" exitCode=0 Nov 03 21:44:27.175276 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:27.171785 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" event={"ID":"0a8d742e-582e-461c-93ce-26b511bfed39","Type":"ContainerDied","Data":"78a7f0e04c5d1c8b59c50b80fe599515aae7e2f4d8947800eb956f3930318ad9"} Nov 03 21:44:27.640041 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:27.640008 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" Nov 03 21:44:27.771598 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:27.770075 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-maas-api-xks-ca\") pod \"58e4dad7-bdde-4909-af0d-4f8eb8f145e5\" (UID: \"58e4dad7-bdde-4909-af0d-4f8eb8f145e5\") " Nov 03 21:44:27.771598 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:27.770152 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-maas-api-openshift-ca\") pod \"58e4dad7-bdde-4909-af0d-4f8eb8f145e5\" (UID: \"58e4dad7-bdde-4909-af0d-4f8eb8f145e5\") " Nov 03 21:44:27.771598 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:27.770400 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-kube-api-access-cwsnc\" (UniqueName: \"kubernetes.io/projected/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-kube-api-access-cwsnc\") pod \"58e4dad7-bdde-4909-af0d-4f8eb8f145e5\" (UID: \"58e4dad7-bdde-4909-af0d-4f8eb8f145e5\") " Nov 03 21:44:27.772413 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:27.772374 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-maas-api-openshift-ca" pod "58e4dad7-bdde-4909-af0d-4f8eb8f145e5" (UID: "58e4dad7-bdde-4909-af0d-4f8eb8f145e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:27.783300 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:27.783240 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-kube-api-access-cwsnc" pod "58e4dad7-bdde-4909-af0d-4f8eb8f145e5" (UID: "58e4dad7-bdde-4909-af0d-4f8eb8f145e5"). InnerVolumeSpecName "kube-api-access-cwsnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:27.792442 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:27.792405 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-maas-api-xks-ca" pod "58e4dad7-bdde-4909-af0d-4f8eb8f145e5" (UID: "58e4dad7-bdde-4909-af0d-4f8eb8f145e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:27.875686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:27.874862 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwsnc\" (UniqueName: \"kubernetes.io/projected/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-kube-api-access-cwsnc\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:27.875686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:27.874914 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:27.875686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:27.874928 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e4dad7-bdde-4909-af0d-4f8eb8f145e5-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:28.353642 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:28.353605 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" Nov 03 21:44:28.357786 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:28.357749 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst" event={"ID":"58e4dad7-bdde-4909-af0d-4f8eb8f145e5","Type":"ContainerDied","Data":"be27a979f1d89bc763ed110eff578b73761b7a1471e70f0839db8f1f6f471039"} Nov 03 21:44:28.358382 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:28.358359 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be27a979f1d89bc763ed110eff578b73761b7a1471e70f0839db8f1f6f471039" Nov 03 21:44:29.203685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:29.203313 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" Nov 03 21:44:29.216342 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:29.215886 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8d742e-582e-461c-93ce-26b511bfed39-kube-api-access-j6gkl\" (UniqueName: \"kubernetes.io/projected/0a8d742e-582e-461c-93ce-26b511bfed39-kube-api-access-j6gkl\") pod \"0a8d742e-582e-461c-93ce-26b511bfed39\" (UID: \"0a8d742e-582e-461c-93ce-26b511bfed39\") " Nov 03 21:44:29.216342 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:29.216137 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8d742e-582e-461c-93ce-26b511bfed39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8d742e-582e-461c-93ce-26b511bfed39-maas-api-openshift-ca\") pod \"0a8d742e-582e-461c-93ce-26b511bfed39\" (UID: \"0a8d742e-582e-461c-93ce-26b511bfed39\") " Nov 03 21:44:29.216342 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:29.216196 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a8d742e-582e-461c-93ce-26b511bfed39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8d742e-582e-461c-93ce-26b511bfed39-maas-api-xks-ca\") pod \"0a8d742e-582e-461c-93ce-26b511bfed39\" (UID: \"0a8d742e-582e-461c-93ce-26b511bfed39\") " Nov 03 21:44:29.222694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:29.217399 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8d742e-582e-461c-93ce-26b511bfed39-maas-api-openshift-ca" pod "0a8d742e-582e-461c-93ce-26b511bfed39" (UID: "0a8d742e-582e-461c-93ce-26b511bfed39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:29.255687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:29.251219 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a8d742e-582e-461c-93ce-26b511bfed39-maas-api-xks-ca" pod "0a8d742e-582e-461c-93ce-26b511bfed39" (UID: "0a8d742e-582e-461c-93ce-26b511bfed39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:29.276732 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:29.270604 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8d742e-582e-461c-93ce-26b511bfed39-kube-api-access-j6gkl" pod "0a8d742e-582e-461c-93ce-26b511bfed39" (UID: "0a8d742e-582e-461c-93ce-26b511bfed39"). InnerVolumeSpecName "kube-api-access-j6gkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:29.330719 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:29.322823 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6gkl\" (UniqueName: \"kubernetes.io/projected/0a8d742e-582e-461c-93ce-26b511bfed39-kube-api-access-j6gkl\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:29.330719 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:29.322858 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8d742e-582e-461c-93ce-26b511bfed39-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:29.330719 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:29.322871 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8d742e-582e-461c-93ce-26b511bfed39-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:29.395691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:29.392350 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" event={"ID":"0a8d742e-582e-461c-93ce-26b511bfed39","Type":"ContainerDied","Data":"4af9f923e47e4177227d12cf8bda5399305add03d9c0ec1b8f262e7df2f75cf3"} Nov 03 21:44:29.395691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:29.392414 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4af9f923e47e4177227d12cf8bda5399305add03d9c0ec1b8f262e7df2f75cf3" Nov 03 21:44:29.395691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:29.392486 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6" Nov 03 21:44:30.860397 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.860356 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr"] Nov 03 21:44:30.861233 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.861209 5041 cpu_manager.go:414] "containerMap: removing container" podUID="58e4dad7-bdde-4909-af0d-4f8eb8f145e5" containerName="revoke-keys" Nov 03 21:44:30.861233 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.861233 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e4dad7-bdde-4909-af0d-4f8eb8f145e5" containerName="revoke-keys" Nov 03 21:44:30.861398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.861259 5041 cpu_manager.go:414] "containerMap: removing container" podUID="0a8d742e-582e-461c-93ce-26b511bfed39" containerName="revoke-keys" Nov 03 21:44:30.861398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.861267 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8d742e-582e-461c-93ce-26b511bfed39" containerName="revoke-keys" Nov 03 21:44:30.861398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.861323 5041 cpu_manager.go:414] "containerMap: removing container" podUID="479af13b-6f89-4a24-9fd5-619888b129cc" containerName="revoke-keys" Nov 03 21:44:30.861398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.861330 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479af13b-6f89-4a24-9fd5-619888b129cc" containerName="revoke-keys" Nov 03 21:44:30.861594 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.861496 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8d742e-582e-461c-93ce-26b511bfed39" containerName="revoke-keys" Nov 03 21:44:30.861594 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.861519 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="479af13b-6f89-4a24-9fd5-619888b129cc" containerName="revoke-keys" Nov 03 21:44:30.861594 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.861538 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e4dad7-bdde-4909-af0d-4f8eb8f145e5" containerName="revoke-keys" Nov 03 21:44:30.865046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.865025 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" Nov 03 21:44:30.883370 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.883339 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr"] Nov 03 21:44:30.891233 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.891206 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:44:30.891390 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.891330 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:44:30.917654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.917617 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fqwr\" (UniqueName: \"kubernetes.io/projected/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-kube-api-access-5fqwr\") pod \"maas-api-revoke-sub-87e8f73e-d77qr\" (UID: \"7ed16bde-a47b-48fd-a876-8836ef1d0ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" Nov 03 21:44:30.917852 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.917745 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87e8f73e-d77qr\" (UID: \"7ed16bde-a47b-48fd-a876-8836ef1d0ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" Nov 03 21:44:30.917852 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:30.917823 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87e8f73e-d77qr\" (UID: \"7ed16bde-a47b-48fd-a876-8836ef1d0ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" Nov 03 21:44:31.018965 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:31.018915 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87e8f73e-d77qr\" (UID: \"7ed16bde-a47b-48fd-a876-8836ef1d0ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" Nov 03 21:44:31.019152 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:31.019066 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqwr\" (UniqueName: \"kubernetes.io/projected/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-kube-api-access-5fqwr\") pod \"maas-api-revoke-sub-87e8f73e-d77qr\" (UID: \"7ed16bde-a47b-48fd-a876-8836ef1d0ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" Nov 03 21:44:31.019206 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:31.019151 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87e8f73e-d77qr\" (UID: \"7ed16bde-a47b-48fd-a876-8836ef1d0ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" Nov 03 21:44:31.019768 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:31.019739 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87e8f73e-d77qr\" (UID: \"7ed16bde-a47b-48fd-a876-8836ef1d0ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" Nov 03 21:44:31.021925 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:31.021896 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87e8f73e-d77qr\" (UID: \"7ed16bde-a47b-48fd-a876-8836ef1d0ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" Nov 03 21:44:31.040362 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:31.040329 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fqwr\" (UniqueName: \"kubernetes.io/projected/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-kube-api-access-5fqwr\") pod \"maas-api-revoke-sub-87e8f73e-d77qr\" (UID: \"7ed16bde-a47b-48fd-a876-8836ef1d0ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" Nov 03 21:44:31.184959 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:31.184877 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" Nov 03 21:44:31.492565 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:31.492527 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr"] Nov 03 21:44:31.492745 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:44:31.492719 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed16bde_a47b_48fd_a876_8836ef1d0ae7.slice/crio-f209a60dc71091fbe4ba889d5453a7fe85ea8507181bc3338519b4dc549de3ef WatchSource:0}: Error finding container f209a60dc71091fbe4ba889d5453a7fe85ea8507181bc3338519b4dc549de3ef: Status 404 returned error can't find the container with id f209a60dc71091fbe4ba889d5453a7fe85ea8507181bc3338519b4dc549de3ef Nov 03 21:44:32.442421 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:32.442380 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" event={"ID":"7ed16bde-a47b-48fd-a876-8836ef1d0ae7","Type":"ContainerStarted","Data":"7e3039c2f39622feb0796bd4d2fac0275ffe5307d7441bdbabeb4b9b88e24ab8"} Nov 03 21:44:32.443855 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:32.442495 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" event={"ID":"7ed16bde-a47b-48fd-a876-8836ef1d0ae7","Type":"ContainerStarted","Data":"f209a60dc71091fbe4ba889d5453a7fe85ea8507181bc3338519b4dc549de3ef"} Nov 03 21:44:32.486360 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:32.486308 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" podStartSLOduration=2.48629456 podStartE2EDuration="2.48629456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:32.482585427 +0000 UTC m=+887.989813708" watchObservedRunningTime="2026-11-03 21:44:32.48629456 +0000 UTC m=+887.993522840" Nov 03 21:44:33.026866 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.026829 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw"] Nov 03 21:44:33.031138 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.031120 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" Nov 03 21:44:33.048138 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.048097 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96d4s\" (UniqueName: \"kubernetes.io/projected/c7c23b00-eba0-44d0-b0ed-5c0300aca453-kube-api-access-96d4s\") pod \"maas-api-revoke-sub-04b00af2-rcdkw\" (UID: \"c7c23b00-eba0-44d0-b0ed-5c0300aca453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" Nov 03 21:44:33.048281 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.048224 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c23b00-eba0-44d0-b0ed-5c0300aca453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04b00af2-rcdkw\" (UID: \"c7c23b00-eba0-44d0-b0ed-5c0300aca453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" Nov 03 21:44:33.048354 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.048311 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw"] Nov 03 21:44:33.048354 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.048335 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c23b00-eba0-44d0-b0ed-5c0300aca453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04b00af2-rcdkw\" (UID: \"c7c23b00-eba0-44d0-b0ed-5c0300aca453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" Nov 03 21:44:33.150320 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.150279 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c23b00-eba0-44d0-b0ed-5c0300aca453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04b00af2-rcdkw\" (UID: \"c7c23b00-eba0-44d0-b0ed-5c0300aca453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" Nov 03 21:44:33.150881 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.150853 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96d4s\" (UniqueName: \"kubernetes.io/projected/c7c23b00-eba0-44d0-b0ed-5c0300aca453-kube-api-access-96d4s\") pod \"maas-api-revoke-sub-04b00af2-rcdkw\" (UID: \"c7c23b00-eba0-44d0-b0ed-5c0300aca453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" Nov 03 21:44:33.151026 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.150926 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c23b00-eba0-44d0-b0ed-5c0300aca453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04b00af2-rcdkw\" (UID: \"c7c23b00-eba0-44d0-b0ed-5c0300aca453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" Nov 03 21:44:33.151693 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.151643 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c23b00-eba0-44d0-b0ed-5c0300aca453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04b00af2-rcdkw\" (UID: \"c7c23b00-eba0-44d0-b0ed-5c0300aca453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" Nov 03 21:44:33.153171 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.153145 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c23b00-eba0-44d0-b0ed-5c0300aca453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04b00af2-rcdkw\" (UID: \"c7c23b00-eba0-44d0-b0ed-5c0300aca453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" Nov 03 21:44:33.172750 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.172704 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96d4s\" (UniqueName: \"kubernetes.io/projected/c7c23b00-eba0-44d0-b0ed-5c0300aca453-kube-api-access-96d4s\") pod \"maas-api-revoke-sub-04b00af2-rcdkw\" (UID: \"c7c23b00-eba0-44d0-b0ed-5c0300aca453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" Nov 03 21:44:33.346138 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.346059 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" Nov 03 21:44:33.453242 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.453199 5041 generic.go:356] "Generic (PLEG): container finished" podID="7ed16bde-a47b-48fd-a876-8836ef1d0ae7" containerID="7e3039c2f39622feb0796bd4d2fac0275ffe5307d7441bdbabeb4b9b88e24ab8" exitCode=0 Nov 03 21:44:33.453624 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.453283 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" event={"ID":"7ed16bde-a47b-48fd-a876-8836ef1d0ae7","Type":"ContainerDied","Data":"7e3039c2f39622feb0796bd4d2fac0275ffe5307d7441bdbabeb4b9b88e24ab8"} Nov 03 21:44:33.494172 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.494135 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv"] Nov 03 21:44:33.499673 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.499632 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" Nov 03 21:44:33.516985 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.516959 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv"] Nov 03 21:44:33.555848 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.555817 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22b84cc7-lqdzv\" (UID: \"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" Nov 03 21:44:33.556017 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.555896 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22b84cc7-lqdzv\" (UID: \"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" Nov 03 21:44:33.556095 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.556079 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k26f\" (UniqueName: \"kubernetes.io/projected/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-kube-api-access-2k26f\") pod \"maas-api-revoke-sub-22b84cc7-lqdzv\" (UID: \"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" Nov 03 21:44:33.651147 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:44:33.651111 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c23b00_eba0_44d0_b0ed_5c0300aca453.slice/crio-5a9475cf030e67230aaf1cd7109fbd7c9bb593024be1ccfa4f6c3e6f0814d786 WatchSource:0}: Error finding container 5a9475cf030e67230aaf1cd7109fbd7c9bb593024be1ccfa4f6c3e6f0814d786: Status 404 returned error can't find the container with id 5a9475cf030e67230aaf1cd7109fbd7c9bb593024be1ccfa4f6c3e6f0814d786 Nov 03 21:44:33.651446 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.651423 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw"] Nov 03 21:44:33.657001 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.656978 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k26f\" (UniqueName: \"kubernetes.io/projected/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-kube-api-access-2k26f\") pod \"maas-api-revoke-sub-22b84cc7-lqdzv\" (UID: \"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" Nov 03 21:44:33.657115 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.657067 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22b84cc7-lqdzv\" (UID: \"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" Nov 03 21:44:33.657115 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.657100 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22b84cc7-lqdzv\" (UID: \"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" Nov 03 21:44:33.658134 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.658109 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22b84cc7-lqdzv\" (UID: \"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" Nov 03 21:44:33.659927 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.659902 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22b84cc7-lqdzv\" (UID: \"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" Nov 03 21:44:33.677544 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.677523 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k26f\" (UniqueName: \"kubernetes.io/projected/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-kube-api-access-2k26f\") pod \"maas-api-revoke-sub-22b84cc7-lqdzv\" (UID: \"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" Nov 03 21:44:33.818565 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:33.818537 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" Nov 03 21:44:34.136046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.136011 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv"] Nov 03 21:44:34.143855 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:44:34.143805 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b76ee6_e687_4fe7_9ad0_67a6bbea2c7a.slice/crio-c18a36464c574712b804e1eeea2e7b8a69b2a4517539dbbe1344473bd4f28f7c WatchSource:0}: Error finding container c18a36464c574712b804e1eeea2e7b8a69b2a4517539dbbe1344473bd4f28f7c: Status 404 returned error can't find the container with id c18a36464c574712b804e1eeea2e7b8a69b2a4517539dbbe1344473bd4f28f7c Nov 03 21:44:34.464035 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.464002 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" event={"ID":"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a","Type":"ContainerStarted","Data":"7b9ba69db9a5332f377eed9e6b450712957548fd6052da1ee03d443fd029b84f"} Nov 03 21:44:34.464496 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.464067 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" event={"ID":"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a","Type":"ContainerStarted","Data":"c18a36464c574712b804e1eeea2e7b8a69b2a4517539dbbe1344473bd4f28f7c"} Nov 03 21:44:34.466132 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.466098 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" event={"ID":"c7c23b00-eba0-44d0-b0ed-5c0300aca453","Type":"ContainerStarted","Data":"a57f6fe153f0327b19ff7da3572c5b0a62723f9d675b99902e51a3786b6c45f6"} Nov 03 21:44:34.466243 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.466155 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" event={"ID":"c7c23b00-eba0-44d0-b0ed-5c0300aca453","Type":"ContainerStarted","Data":"5a9475cf030e67230aaf1cd7109fbd7c9bb593024be1ccfa4f6c3e6f0814d786"} Nov 03 21:44:34.503672 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.503609 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" podStartSLOduration=1.503594519 podStartE2EDuration="1.503594519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:34.503015735 +0000 UTC m=+890.010244016" watchObservedRunningTime="2026-11-03 21:44:34.503594519 +0000 UTC m=+890.010822799" Nov 03 21:44:34.555922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.552467 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" podStartSLOduration=1.55244765 podStartE2EDuration="1.55244765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:34.545776862 +0000 UTC m=+890.053005166" watchObservedRunningTime="2026-11-03 21:44:34.55244765 +0000 UTC m=+890.059675931" Nov 03 21:44:34.647533 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.647498 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" Nov 03 21:44:34.669447 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.669413 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-maas-api-openshift-ca\") pod \"7ed16bde-a47b-48fd-a876-8836ef1d0ae7\" (UID: \"7ed16bde-a47b-48fd-a876-8836ef1d0ae7\") " Nov 03 21:44:34.669708 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.669653 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-kube-api-access-5fqwr\" (UniqueName: \"kubernetes.io/projected/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-kube-api-access-5fqwr\") pod \"7ed16bde-a47b-48fd-a876-8836ef1d0ae7\" (UID: \"7ed16bde-a47b-48fd-a876-8836ef1d0ae7\") " Nov 03 21:44:34.669775 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.669763 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-maas-api-xks-ca\") pod \"7ed16bde-a47b-48fd-a876-8836ef1d0ae7\" (UID: \"7ed16bde-a47b-48fd-a876-8836ef1d0ae7\") " Nov 03 21:44:34.669860 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.669838 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-maas-api-openshift-ca" pod "7ed16bde-a47b-48fd-a876-8836ef1d0ae7" (UID: "7ed16bde-a47b-48fd-a876-8836ef1d0ae7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:34.670384 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.670320 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:34.671803 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.671776 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-kube-api-access-5fqwr" pod "7ed16bde-a47b-48fd-a876-8836ef1d0ae7" (UID: "7ed16bde-a47b-48fd-a876-8836ef1d0ae7"). InnerVolumeSpecName "kube-api-access-5fqwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:34.672020 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.671994 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-maas-api-xks-ca" pod "7ed16bde-a47b-48fd-a876-8836ef1d0ae7" (UID: "7ed16bde-a47b-48fd-a876-8836ef1d0ae7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:34.771637 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.771548 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:34.771637 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.771587 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fqwr\" (UniqueName: \"kubernetes.io/projected/7ed16bde-a47b-48fd-a876-8836ef1d0ae7-kube-api-access-5fqwr\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:34.953745 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.953709 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn"] Nov 03 21:44:34.960364 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.954974 5041 cpu_manager.go:414] "containerMap: removing container" podUID="7ed16bde-a47b-48fd-a876-8836ef1d0ae7" containerName="revoke-keys" Nov 03 21:44:34.960364 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.955009 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed16bde-a47b-48fd-a876-8836ef1d0ae7" containerName="revoke-keys" Nov 03 21:44:34.960364 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.955501 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed16bde-a47b-48fd-a876-8836ef1d0ae7" containerName="revoke-keys" Nov 03 21:44:34.966393 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.965697 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" Nov 03 21:44:34.976416 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:34.976283 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn"] Nov 03 21:44:35.075136 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.075057 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6a7e516-8xgfn\" (UID: \"6fa15f20-4f4b-4877-b1c9-84014f0dc01b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" Nov 03 21:44:35.075136 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.075114 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6a7e516-8xgfn\" (UID: \"6fa15f20-4f4b-4877-b1c9-84014f0dc01b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" Nov 03 21:44:35.075327 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.075204 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td88g\" (UniqueName: \"kubernetes.io/projected/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-kube-api-access-td88g\") pod \"maas-api-revoke-sub-f6a7e516-8xgfn\" (UID: \"6fa15f20-4f4b-4877-b1c9-84014f0dc01b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" Nov 03 21:44:35.176888 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.176853 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6a7e516-8xgfn\" (UID: \"6fa15f20-4f4b-4877-b1c9-84014f0dc01b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" Nov 03 21:44:35.177065 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.176911 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6a7e516-8xgfn\" (UID: \"6fa15f20-4f4b-4877-b1c9-84014f0dc01b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" Nov 03 21:44:35.177065 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.176938 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td88g\" (UniqueName: \"kubernetes.io/projected/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-kube-api-access-td88g\") pod \"maas-api-revoke-sub-f6a7e516-8xgfn\" (UID: \"6fa15f20-4f4b-4877-b1c9-84014f0dc01b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" Nov 03 21:44:35.177579 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.177551 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6a7e516-8xgfn\" (UID: \"6fa15f20-4f4b-4877-b1c9-84014f0dc01b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" Nov 03 21:44:35.179224 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.179205 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6a7e516-8xgfn\" (UID: \"6fa15f20-4f4b-4877-b1c9-84014f0dc01b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" Nov 03 21:44:35.197287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.197260 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td88g\" (UniqueName: \"kubernetes.io/projected/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-kube-api-access-td88g\") pod \"maas-api-revoke-sub-f6a7e516-8xgfn\" (UID: \"6fa15f20-4f4b-4877-b1c9-84014f0dc01b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" Nov 03 21:44:35.284357 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.284318 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" Nov 03 21:44:35.477078 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.477038 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" event={"ID":"7ed16bde-a47b-48fd-a876-8836ef1d0ae7","Type":"ContainerDied","Data":"f209a60dc71091fbe4ba889d5453a7fe85ea8507181bc3338519b4dc549de3ef"} Nov 03 21:44:35.477078 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.477062 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr" Nov 03 21:44:35.477536 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.477106 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f209a60dc71091fbe4ba889d5453a7fe85ea8507181bc3338519b4dc549de3ef" Nov 03 21:44:35.479018 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.478993 5041 generic.go:356] "Generic (PLEG): container finished" podID="25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a" containerID="7b9ba69db9a5332f377eed9e6b450712957548fd6052da1ee03d443fd029b84f" exitCode=0 Nov 03 21:44:35.479418 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.479082 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" event={"ID":"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a","Type":"ContainerDied","Data":"7b9ba69db9a5332f377eed9e6b450712957548fd6052da1ee03d443fd029b84f"} Nov 03 21:44:35.480917 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.480891 5041 generic.go:356] "Generic (PLEG): container finished" podID="c7c23b00-eba0-44d0-b0ed-5c0300aca453" containerID="a57f6fe153f0327b19ff7da3572c5b0a62723f9d675b99902e51a3786b6c45f6" exitCode=0 Nov 03 21:44:35.481013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.480965 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" event={"ID":"c7c23b00-eba0-44d0-b0ed-5c0300aca453","Type":"ContainerDied","Data":"a57f6fe153f0327b19ff7da3572c5b0a62723f9d675b99902e51a3786b6c45f6"} Nov 03 21:44:35.591101 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:35.591033 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn"] Nov 03 21:44:35.591399 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:44:35.591378 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa15f20_4f4b_4877_b1c9_84014f0dc01b.slice/crio-67078684a5d6df5cd6b9aa58cd97d49315e59dda6a0e13165ff05bcbbfc2390e WatchSource:0}: Error finding container 67078684a5d6df5cd6b9aa58cd97d49315e59dda6a0e13165ff05bcbbfc2390e: Status 404 returned error can't find the container with id 67078684a5d6df5cd6b9aa58cd97d49315e59dda6a0e13165ff05bcbbfc2390e Nov 03 21:44:36.492249 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.492205 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" event={"ID":"6fa15f20-4f4b-4877-b1c9-84014f0dc01b","Type":"ContainerStarted","Data":"54a8fb4b1849aa90bb45f12ac447bc281352e6f5393958bfdabc658e893567c7"} Nov 03 21:44:36.492769 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.492297 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" event={"ID":"6fa15f20-4f4b-4877-b1c9-84014f0dc01b","Type":"ContainerStarted","Data":"67078684a5d6df5cd6b9aa58cd97d49315e59dda6a0e13165ff05bcbbfc2390e"} Nov 03 21:44:36.552898 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.552849 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" podStartSLOduration=2.552834392 podStartE2EDuration="2.552834392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:36.536133704 +0000 UTC m=+892.043362003" watchObservedRunningTime="2026-11-03 21:44:36.552834392 +0000 UTC m=+892.060062672" Nov 03 21:44:36.731562 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.731532 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" Nov 03 21:44:36.737548 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.737521 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" Nov 03 21:44:36.801442 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.801360 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c23b00-eba0-44d0-b0ed-5c0300aca453-kube-api-access-96d4s\" (UniqueName: \"kubernetes.io/projected/c7c23b00-eba0-44d0-b0ed-5c0300aca453-kube-api-access-96d4s\") pod \"c7c23b00-eba0-44d0-b0ed-5c0300aca453\" (UID: \"c7c23b00-eba0-44d0-b0ed-5c0300aca453\") " Nov 03 21:44:36.801442 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.801402 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-maas-api-openshift-ca\") pod \"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a\" (UID: \"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a\") " Nov 03 21:44:36.801442 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.801431 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c23b00-eba0-44d0-b0ed-5c0300aca453-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c23b00-eba0-44d0-b0ed-5c0300aca453-maas-api-openshift-ca\") pod \"c7c23b00-eba0-44d0-b0ed-5c0300aca453\" (UID: \"c7c23b00-eba0-44d0-b0ed-5c0300aca453\") " Nov 03 21:44:36.801777 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.801464 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7c23b00-eba0-44d0-b0ed-5c0300aca453-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c23b00-eba0-44d0-b0ed-5c0300aca453-maas-api-xks-ca\") pod \"c7c23b00-eba0-44d0-b0ed-5c0300aca453\" (UID: \"c7c23b00-eba0-44d0-b0ed-5c0300aca453\") " Nov 03 21:44:36.801777 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.801688 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-maas-api-xks-ca\") pod \"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a\" (UID: \"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a\") " Nov 03 21:44:36.801777 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.801716 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-kube-api-access-2k26f\" (UniqueName: \"kubernetes.io/projected/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-kube-api-access-2k26f\") pod \"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a\" (UID: \"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a\") " Nov 03 21:44:36.802364 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.802292 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c23b00-eba0-44d0-b0ed-5c0300aca453-maas-api-openshift-ca" pod "c7c23b00-eba0-44d0-b0ed-5c0300aca453" (UID: "c7c23b00-eba0-44d0-b0ed-5c0300aca453"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:36.802901 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.802880 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-maas-api-openshift-ca" pod "25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a" (UID: "25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:36.804881 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.804853 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c23b00-eba0-44d0-b0ed-5c0300aca453-kube-api-access-96d4s" pod "c7c23b00-eba0-44d0-b0ed-5c0300aca453" (UID: "c7c23b00-eba0-44d0-b0ed-5c0300aca453"). InnerVolumeSpecName "kube-api-access-96d4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:36.804952 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.804918 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c23b00-eba0-44d0-b0ed-5c0300aca453-maas-api-xks-ca" pod "c7c23b00-eba0-44d0-b0ed-5c0300aca453" (UID: "c7c23b00-eba0-44d0-b0ed-5c0300aca453"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:36.805954 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.805930 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-maas-api-xks-ca" pod "25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a" (UID: "25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:36.807120 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.807094 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-kube-api-access-2k26f" pod "25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a" (UID: "25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a"). InnerVolumeSpecName "kube-api-access-2k26f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:36.903677 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.903632 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:36.903859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.903701 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k26f\" (UniqueName: \"kubernetes.io/projected/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-kube-api-access-2k26f\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:36.903859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.903719 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96d4s\" (UniqueName: \"kubernetes.io/projected/c7c23b00-eba0-44d0-b0ed-5c0300aca453-kube-api-access-96d4s\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:36.903859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.903733 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:36.903859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.903748 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c23b00-eba0-44d0-b0ed-5c0300aca453-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:36.903859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:36.903765 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c23b00-eba0-44d0-b0ed-5c0300aca453-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:37.055038 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.054963 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2"] Nov 03 21:44:37.055581 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.055565 5041 cpu_manager.go:414] "containerMap: removing container" podUID="25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a" containerName="revoke-keys" Nov 03 21:44:37.055630 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.055583 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a" containerName="revoke-keys" Nov 03 21:44:37.055630 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.055616 5041 cpu_manager.go:414] "containerMap: removing container" podUID="c7c23b00-eba0-44d0-b0ed-5c0300aca453" containerName="revoke-keys" Nov 03 21:44:37.055630 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.055622 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c23b00-eba0-44d0-b0ed-5c0300aca453" containerName="revoke-keys" Nov 03 21:44:37.055739 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.055721 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c23b00-eba0-44d0-b0ed-5c0300aca453" containerName="revoke-keys" Nov 03 21:44:37.055739 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.055735 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a" containerName="revoke-keys" Nov 03 21:44:37.059484 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.059464 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" Nov 03 21:44:37.078073 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.078043 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2"] Nov 03 21:44:37.105801 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.105765 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t47rt\" (UniqueName: \"kubernetes.io/projected/3fa31e39-9530-4a6b-acaf-7e9eaf014929-kube-api-access-t47rt\") pod \"maas-api-revoke-sub-8f4b2ad8-crqp2\" (UID: \"3fa31e39-9530-4a6b-acaf-7e9eaf014929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" Nov 03 21:44:37.105987 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.105812 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa31e39-9530-4a6b-acaf-7e9eaf014929-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f4b2ad8-crqp2\" (UID: \"3fa31e39-9530-4a6b-acaf-7e9eaf014929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" Nov 03 21:44:37.106059 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.106013 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa31e39-9530-4a6b-acaf-7e9eaf014929-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f4b2ad8-crqp2\" (UID: \"3fa31e39-9530-4a6b-acaf-7e9eaf014929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" Nov 03 21:44:37.207169 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.207122 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa31e39-9530-4a6b-acaf-7e9eaf014929-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f4b2ad8-crqp2\" (UID: \"3fa31e39-9530-4a6b-acaf-7e9eaf014929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" Nov 03 21:44:37.207355 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.207263 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t47rt\" (UniqueName: \"kubernetes.io/projected/3fa31e39-9530-4a6b-acaf-7e9eaf014929-kube-api-access-t47rt\") pod \"maas-api-revoke-sub-8f4b2ad8-crqp2\" (UID: \"3fa31e39-9530-4a6b-acaf-7e9eaf014929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" Nov 03 21:44:37.207355 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.207299 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa31e39-9530-4a6b-acaf-7e9eaf014929-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f4b2ad8-crqp2\" (UID: \"3fa31e39-9530-4a6b-acaf-7e9eaf014929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" Nov 03 21:44:37.208051 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.208031 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa31e39-9530-4a6b-acaf-7e9eaf014929-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f4b2ad8-crqp2\" (UID: \"3fa31e39-9530-4a6b-acaf-7e9eaf014929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" Nov 03 21:44:37.209944 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.209917 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa31e39-9530-4a6b-acaf-7e9eaf014929-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f4b2ad8-crqp2\" (UID: \"3fa31e39-9530-4a6b-acaf-7e9eaf014929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" Nov 03 21:44:37.227601 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.227575 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t47rt\" (UniqueName: \"kubernetes.io/projected/3fa31e39-9530-4a6b-acaf-7e9eaf014929-kube-api-access-t47rt\") pod \"maas-api-revoke-sub-8f4b2ad8-crqp2\" (UID: \"3fa31e39-9530-4a6b-acaf-7e9eaf014929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" Nov 03 21:44:37.375339 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.375258 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" Nov 03 21:44:37.504953 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.504919 5041 generic.go:356] "Generic (PLEG): container finished" podID="6fa15f20-4f4b-4877-b1c9-84014f0dc01b" containerID="54a8fb4b1849aa90bb45f12ac447bc281352e6f5393958bfdabc658e893567c7" exitCode=0 Nov 03 21:44:37.505685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.505010 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" event={"ID":"6fa15f20-4f4b-4877-b1c9-84014f0dc01b","Type":"ContainerDied","Data":"54a8fb4b1849aa90bb45f12ac447bc281352e6f5393958bfdabc658e893567c7"} Nov 03 21:44:37.506718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.506699 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" Nov 03 21:44:37.506839 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.506709 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv" event={"ID":"25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a","Type":"ContainerDied","Data":"c18a36464c574712b804e1eeea2e7b8a69b2a4517539dbbe1344473bd4f28f7c"} Nov 03 21:44:37.506839 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.506759 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c18a36464c574712b804e1eeea2e7b8a69b2a4517539dbbe1344473bd4f28f7c" Nov 03 21:44:37.508414 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.508378 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" Nov 03 21:44:37.508414 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.508403 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw" event={"ID":"c7c23b00-eba0-44d0-b0ed-5c0300aca453","Type":"ContainerDied","Data":"5a9475cf030e67230aaf1cd7109fbd7c9bb593024be1ccfa4f6c3e6f0814d786"} Nov 03 21:44:37.508550 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.508445 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a9475cf030e67230aaf1cd7109fbd7c9bb593024be1ccfa4f6c3e6f0814d786" Nov 03 21:44:37.680809 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:37.680771 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2"] Nov 03 21:44:37.681034 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:44:37.681004 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa31e39_9530_4a6b_acaf_7e9eaf014929.slice/crio-68584d23504bd1d10f5e20efad2a9d1f3813bc5b8ec6f1a353393de983b2a035 WatchSource:0}: Error finding container 68584d23504bd1d10f5e20efad2a9d1f3813bc5b8ec6f1a353393de983b2a035: Status 404 returned error can't find the container with id 68584d23504bd1d10f5e20efad2a9d1f3813bc5b8ec6f1a353393de983b2a035 Nov 03 21:44:38.520776 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:38.520733 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" event={"ID":"3fa31e39-9530-4a6b-acaf-7e9eaf014929","Type":"ContainerStarted","Data":"f2c59208cd16c4713aeb0dd988641ecb74d7f24c9c99eb768dbf6ebe61c00e1c"} Nov 03 21:44:38.521321 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:38.520804 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" event={"ID":"3fa31e39-9530-4a6b-acaf-7e9eaf014929","Type":"ContainerStarted","Data":"68584d23504bd1d10f5e20efad2a9d1f3813bc5b8ec6f1a353393de983b2a035"} Nov 03 21:44:38.565671 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:38.565605 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" podStartSLOduration=1.565591533 podStartE2EDuration="1.565591533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:38.561958079 +0000 UTC m=+894.069186391" watchObservedRunningTime="2026-11-03 21:44:38.565591533 +0000 UTC m=+894.072819850" Nov 03 21:44:38.699121 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:38.699093 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" Nov 03 21:44:38.727220 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:38.727195 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-maas-api-openshift-ca\") pod \"6fa15f20-4f4b-4877-b1c9-84014f0dc01b\" (UID: \"6fa15f20-4f4b-4877-b1c9-84014f0dc01b\") " Nov 03 21:44:38.727358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:38.727292 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-kube-api-access-td88g\" (UniqueName: \"kubernetes.io/projected/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-kube-api-access-td88g\") pod \"6fa15f20-4f4b-4877-b1c9-84014f0dc01b\" (UID: \"6fa15f20-4f4b-4877-b1c9-84014f0dc01b\") " Nov 03 21:44:38.727519 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:38.727466 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-maas-api-xks-ca\") pod \"6fa15f20-4f4b-4877-b1c9-84014f0dc01b\" (UID: \"6fa15f20-4f4b-4877-b1c9-84014f0dc01b\") " Nov 03 21:44:38.727619 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:38.727591 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-maas-api-openshift-ca" pod "6fa15f20-4f4b-4877-b1c9-84014f0dc01b" (UID: "6fa15f20-4f4b-4877-b1c9-84014f0dc01b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:38.727978 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:38.727959 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:38.729782 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:38.729760 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-kube-api-access-td88g" pod "6fa15f20-4f4b-4877-b1c9-84014f0dc01b" (UID: "6fa15f20-4f4b-4877-b1c9-84014f0dc01b"). InnerVolumeSpecName "kube-api-access-td88g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:38.729994 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:38.729979 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-maas-api-xks-ca" pod "6fa15f20-4f4b-4877-b1c9-84014f0dc01b" (UID: "6fa15f20-4f4b-4877-b1c9-84014f0dc01b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:38.829388 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:38.829349 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td88g\" (UniqueName: \"kubernetes.io/projected/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-kube-api-access-td88g\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:38.829388 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:38.829389 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fa15f20-4f4b-4877-b1c9-84014f0dc01b-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:39.544718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:39.541591 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" Nov 03 21:44:39.544718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:39.541618 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn" event={"ID":"6fa15f20-4f4b-4877-b1c9-84014f0dc01b","Type":"ContainerDied","Data":"67078684a5d6df5cd6b9aa58cd97d49315e59dda6a0e13165ff05bcbbfc2390e"} Nov 03 21:44:39.544718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:39.541705 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67078684a5d6df5cd6b9aa58cd97d49315e59dda6a0e13165ff05bcbbfc2390e" Nov 03 21:44:39.544718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:39.543993 5041 generic.go:356] "Generic (PLEG): container finished" podID="3fa31e39-9530-4a6b-acaf-7e9eaf014929" containerID="f2c59208cd16c4713aeb0dd988641ecb74d7f24c9c99eb768dbf6ebe61c00e1c" exitCode=0 Nov 03 21:44:39.544718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:39.544086 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" event={"ID":"3fa31e39-9530-4a6b-acaf-7e9eaf014929","Type":"ContainerDied","Data":"f2c59208cd16c4713aeb0dd988641ecb74d7f24c9c99eb768dbf6ebe61c00e1c"} Nov 03 21:44:40.222232 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.222199 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l"] Nov 03 21:44:40.222876 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.222846 5041 cpu_manager.go:414] "containerMap: removing container" podUID="6fa15f20-4f4b-4877-b1c9-84014f0dc01b" containerName="revoke-keys" Nov 03 21:44:40.222876 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.222876 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa15f20-4f4b-4877-b1c9-84014f0dc01b" containerName="revoke-keys" Nov 03 21:44:40.223037 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.222989 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa15f20-4f4b-4877-b1c9-84014f0dc01b" containerName="revoke-keys" Nov 03 21:44:40.226921 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.226901 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.245793 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.245733 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-0a6da1-openshift-default-dockercfg-dhxln\"" Nov 03 21:44:40.264070 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.264038 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l"] Nov 03 21:44:40.349446 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.349397 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-credential-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.349631 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.349514 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj8br\" (UniqueName: \"kubernetes.io/projected/72a848ed-b60e-4690-8933-8abdcac1d8ba-kube-api-access-pj8br\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.349631 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.349571 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-token\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.349631 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.349594 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-envoy\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.349631 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.349619 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-workload-certs\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.349896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.349727 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a848ed-b60e-4690-8933-8abdcac1d8ba-istiod-ca-cert\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.349896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.349750 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-podinfo\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.349896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.349776 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-workload-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.349896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.349793 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-data\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.451330 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.451282 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-token\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.451330 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.451331 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-envoy\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.451590 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.451371 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-workload-certs\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.451590 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.451435 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a848ed-b60e-4690-8933-8abdcac1d8ba-istiod-ca-cert\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.451590 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.451467 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-podinfo\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.451590 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.451505 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-workload-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.451590 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.451528 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-data\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.451861 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.451747 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-credential-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.451861 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.451841 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj8br\" (UniqueName: \"kubernetes.io/projected/72a848ed-b60e-4690-8933-8abdcac1d8ba-kube-api-access-pj8br\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.451959 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.451920 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-workload-certs\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.452164 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.452139 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-data\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.452256 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.452192 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-credential-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.452316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.452255 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-workload-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.452458 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.452431 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a848ed-b60e-4690-8933-8abdcac1d8ba-istiod-ca-cert\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.454096 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.454069 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-envoy\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.454893 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.454866 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-podinfo\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.472458 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.472404 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-token\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.472592 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.472575 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj8br\" (UniqueName: \"kubernetes.io/projected/72a848ed-b60e-4690-8933-8abdcac1d8ba-kube-api-access-pj8br\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.542486 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.542452 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:40.754125 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.754092 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" Nov 03 21:44:40.778310 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:44:40.778274 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:44:40.861353 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.861318 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa31e39-9530-4a6b-acaf-7e9eaf014929-kube-api-access-t47rt\" (UniqueName: \"kubernetes.io/projected/3fa31e39-9530-4a6b-acaf-7e9eaf014929-kube-api-access-t47rt\") pod \"3fa31e39-9530-4a6b-acaf-7e9eaf014929\" (UID: \"3fa31e39-9530-4a6b-acaf-7e9eaf014929\") " Nov 03 21:44:40.861518 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.861375 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa31e39-9530-4a6b-acaf-7e9eaf014929-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa31e39-9530-4a6b-acaf-7e9eaf014929-maas-api-openshift-ca\") pod \"3fa31e39-9530-4a6b-acaf-7e9eaf014929\" (UID: \"3fa31e39-9530-4a6b-acaf-7e9eaf014929\") " Nov 03 21:44:40.861566 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.861516 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa31e39-9530-4a6b-acaf-7e9eaf014929-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa31e39-9530-4a6b-acaf-7e9eaf014929-maas-api-xks-ca\") pod \"3fa31e39-9530-4a6b-acaf-7e9eaf014929\" (UID: \"3fa31e39-9530-4a6b-acaf-7e9eaf014929\") " Nov 03 21:44:40.861803 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.861763 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa31e39-9530-4a6b-acaf-7e9eaf014929-maas-api-openshift-ca" pod "3fa31e39-9530-4a6b-acaf-7e9eaf014929" (UID: "3fa31e39-9530-4a6b-acaf-7e9eaf014929"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:44:40.862061 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.862039 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa31e39-9530-4a6b-acaf-7e9eaf014929-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:40.864084 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.864058 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa31e39-9530-4a6b-acaf-7e9eaf014929-maas-api-xks-ca" pod "3fa31e39-9530-4a6b-acaf-7e9eaf014929" (UID: "3fa31e39-9530-4a6b-acaf-7e9eaf014929"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:44:40.864182 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.864061 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa31e39-9530-4a6b-acaf-7e9eaf014929-kube-api-access-t47rt" pod "3fa31e39-9530-4a6b-acaf-7e9eaf014929" (UID: "3fa31e39-9530-4a6b-acaf-7e9eaf014929"). InnerVolumeSpecName "kube-api-access-t47rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:44:40.868309 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.868287 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l"] Nov 03 21:44:40.869526 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:44:40.869491 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a848ed_b60e_4690_8933_8abdcac1d8ba.slice/crio-4ec065c96a29762e97443864f8d94d201a174da9ee735e3a5e2ee57ec53e7b8b WatchSource:0}: Error finding container 4ec065c96a29762e97443864f8d94d201a174da9ee735e3a5e2ee57ec53e7b8b: Status 404 returned error can't find the container with id 4ec065c96a29762e97443864f8d94d201a174da9ee735e3a5e2ee57ec53e7b8b Nov 03 21:44:40.872114 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.872084 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:44:40.872168 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.872154 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:44:40.872209 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.872183 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:44:40.962831 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.962805 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa31e39-9530-4a6b-acaf-7e9eaf014929-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:40.962831 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:40.962831 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t47rt\" (UniqueName: \"kubernetes.io/projected/3fa31e39-9530-4a6b-acaf-7e9eaf014929-kube-api-access-t47rt\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:44:41.561177 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:41.561147 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" Nov 03 21:44:41.561353 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:41.561150 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2" event={"ID":"3fa31e39-9530-4a6b-acaf-7e9eaf014929","Type":"ContainerDied","Data":"68584d23504bd1d10f5e20efad2a9d1f3813bc5b8ec6f1a353393de983b2a035"} Nov 03 21:44:41.561353 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:41.561281 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68584d23504bd1d10f5e20efad2a9d1f3813bc5b8ec6f1a353393de983b2a035" Nov 03 21:44:41.563143 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:41.563116 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" event={"ID":"72a848ed-b60e-4690-8933-8abdcac1d8ba","Type":"ContainerStarted","Data":"ffc32cf50a5f2f5ef2e5f0637b80b10ccb94d6118c77b1d4d48f766c26b2537a"} Nov 03 21:44:41.563273 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:41.563162 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" event={"ID":"72a848ed-b60e-4690-8933-8abdcac1d8ba","Type":"ContainerStarted","Data":"4ec065c96a29762e97443864f8d94d201a174da9ee735e3a5e2ee57ec53e7b8b"} Nov 03 21:44:41.612086 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:41.612016 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" podStartSLOduration=1.611999011 podStartE2EDuration="1.611999011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:41.607330512 +0000 UTC m=+897.114558793" watchObservedRunningTime="2026-11-03 21:44:41.611999011 +0000 UTC m=+897.119227292" Nov 03 21:44:42.543327 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:42.543292 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:42.545274 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:42.545250 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 21:44:42.545411 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:42.545306 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" podUID="72a848ed-b60e-4690-8933-8abdcac1d8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 21:44:43.556690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:43.545977 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 21:44:43.556690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:43.546034 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" podUID="72a848ed-b60e-4690-8933-8abdcac1d8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 21:44:44.545105 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:44.545065 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 21:44:44.545857 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:44.545824 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" podUID="72a848ed-b60e-4690-8933-8abdcac1d8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 21:44:44.987686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:44.979719 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:44:45.078708 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:45.074743 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:44:45.153738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:45.151247 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:44:45.183837 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:45.183769 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:44:45.546475 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:45.546435 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 21:44:45.546687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:45.546497 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" podUID="72a848ed-b60e-4690-8933-8abdcac1d8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 21:44:45.583783 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:45.583753 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 21:44:45.584321 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:45.584301 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 21:44:46.543332 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:46.543284 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 21:44:46.543786 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:46.543370 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" podUID="72a848ed-b60e-4690-8933-8abdcac1d8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 21:44:47.545119 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:47.545076 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 21:44:47.545556 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:47.545164 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" podUID="72a848ed-b60e-4690-8933-8abdcac1d8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 21:44:48.543168 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:48.543127 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 21:44:48.543330 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:48.543200 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" podUID="72a848ed-b60e-4690-8933-8abdcac1d8ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 21:44:49.748127 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:49.748094 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:49.748649 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:49.748621 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:44:49.749772 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:44:49.749749 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:45:00.156272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.156232 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs"] Nov 03 21:45:00.157140 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.157115 5041 cpu_manager.go:414] "containerMap: removing container" podUID="3fa31e39-9530-4a6b-acaf-7e9eaf014929" containerName="revoke-keys" Nov 03 21:45:00.157237 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.157139 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa31e39-9530-4a6b-acaf-7e9eaf014929" containerName="revoke-keys" Nov 03 21:45:00.157343 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.157329 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa31e39-9530-4a6b-acaf-7e9eaf014929" containerName="revoke-keys" Nov 03 21:45:00.160859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.160836 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229"] Nov 03 21:45:00.160975 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.160944 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" Nov 03 21:45:00.165107 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.165084 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k"] Nov 03 21:45:00.165221 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.165194 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" Nov 03 21:45:00.169494 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.169473 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" Nov 03 21:45:00.170814 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.170788 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl"] Nov 03 21:45:00.181314 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.179728 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" Nov 03 21:45:00.189907 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.186121 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7"] Nov 03 21:45:00.189907 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.186847 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:45:00.200038 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.200010 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs"] Nov 03 21:45:00.200196 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.200148 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" Nov 03 21:45:00.203127 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.203104 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229"] Nov 03 21:45:00.240054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.239994 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k"] Nov 03 21:45:00.248077 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.248044 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl"] Nov 03 21:45:00.265762 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.265733 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7"] Nov 03 21:45:00.334319 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.334285 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4096e-7ea0-4377-a8a0-67c3c27d0036-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs\" (UID: \"c2b4096e-7ea0-4377-a8a0-67c3c27d0036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" Nov 03 21:45:00.334502 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.334344 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcp8n\" (UniqueName: \"kubernetes.io/projected/f43eed4a-92c9-4074-936f-b48712db3dfc-kube-api-access-bcp8n\") pod \"maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7\" (UID: \"f43eed4a-92c9-4074-936f-b48712db3dfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" Nov 03 21:45:00.334502 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.334464 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2rf8\" (UniqueName: \"kubernetes.io/projected/64e997e7-9d8b-469b-ac1f-743fa96f7e1f-kube-api-access-b2rf8\") pod \"maas-api-key-cleanup-29895705-8ct6k\" (UID: \"64e997e7-9d8b-469b-ac1f-743fa96f7e1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" Nov 03 21:45:00.334577 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.334515 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bd26\" (UniqueName: \"kubernetes.io/projected/c2b4096e-7ea0-4377-a8a0-67c3c27d0036-kube-api-access-4bd26\") pod \"maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs\" (UID: \"c2b4096e-7ea0-4377-a8a0-67c3c27d0036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" Nov 03 21:45:00.334615 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.334577 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d94ns\" (UniqueName: \"kubernetes.io/projected/832c2242-e330-4541-961b-4acde0fbd38f-kube-api-access-d94ns\") pod \"maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229\" (UID: \"832c2242-e330-4541-961b-4acde0fbd38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" Nov 03 21:45:00.334701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.334623 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/947188bc-8b80-416c-b31a-c3ed370dd012-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl\" (UID: \"947188bc-8b80-416c-b31a-c3ed370dd012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" Nov 03 21:45:00.334828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.334806 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/832c2242-e330-4541-961b-4acde0fbd38f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229\" (UID: \"832c2242-e330-4541-961b-4acde0fbd38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" Nov 03 21:45:00.334915 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.334842 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43eed4a-92c9-4074-936f-b48712db3dfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7\" (UID: \"f43eed4a-92c9-4074-936f-b48712db3dfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" Nov 03 21:45:00.334915 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.334882 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e997e7-9d8b-469b-ac1f-743fa96f7e1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895705-8ct6k\" (UID: \"64e997e7-9d8b-469b-ac1f-743fa96f7e1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" Nov 03 21:45:00.335011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.334990 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h24n9\" (UniqueName: \"kubernetes.io/projected/947188bc-8b80-416c-b31a-c3ed370dd012-kube-api-access-h24n9\") pod \"maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl\" (UID: \"947188bc-8b80-416c-b31a-c3ed370dd012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" Nov 03 21:45:00.435881 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.435793 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/832c2242-e330-4541-961b-4acde0fbd38f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229\" (UID: \"832c2242-e330-4541-961b-4acde0fbd38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" Nov 03 21:45:00.435881 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.435830 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43eed4a-92c9-4074-936f-b48712db3dfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7\" (UID: \"f43eed4a-92c9-4074-936f-b48712db3dfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" Nov 03 21:45:00.435881 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.435857 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e997e7-9d8b-469b-ac1f-743fa96f7e1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895705-8ct6k\" (UID: \"64e997e7-9d8b-469b-ac1f-743fa96f7e1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" Nov 03 21:45:00.436153 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.435891 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h24n9\" (UniqueName: \"kubernetes.io/projected/947188bc-8b80-416c-b31a-c3ed370dd012-kube-api-access-h24n9\") pod \"maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl\" (UID: \"947188bc-8b80-416c-b31a-c3ed370dd012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" Nov 03 21:45:00.436153 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.435948 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4096e-7ea0-4377-a8a0-67c3c27d0036-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs\" (UID: \"c2b4096e-7ea0-4377-a8a0-67c3c27d0036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" Nov 03 21:45:00.436153 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.436005 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcp8n\" (UniqueName: \"kubernetes.io/projected/f43eed4a-92c9-4074-936f-b48712db3dfc-kube-api-access-bcp8n\") pod \"maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7\" (UID: \"f43eed4a-92c9-4074-936f-b48712db3dfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" Nov 03 21:45:00.436153 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.436083 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2rf8\" (UniqueName: \"kubernetes.io/projected/64e997e7-9d8b-469b-ac1f-743fa96f7e1f-kube-api-access-b2rf8\") pod \"maas-api-key-cleanup-29895705-8ct6k\" (UID: \"64e997e7-9d8b-469b-ac1f-743fa96f7e1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" Nov 03 21:45:00.436153 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.436150 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bd26\" (UniqueName: \"kubernetes.io/projected/c2b4096e-7ea0-4377-a8a0-67c3c27d0036-kube-api-access-4bd26\") pod \"maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs\" (UID: \"c2b4096e-7ea0-4377-a8a0-67c3c27d0036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" Nov 03 21:45:00.436421 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.436180 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d94ns\" (UniqueName: \"kubernetes.io/projected/832c2242-e330-4541-961b-4acde0fbd38f-kube-api-access-d94ns\") pod \"maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229\" (UID: \"832c2242-e330-4541-961b-4acde0fbd38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" Nov 03 21:45:00.436421 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.436216 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/947188bc-8b80-416c-b31a-c3ed370dd012-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl\" (UID: \"947188bc-8b80-416c-b31a-c3ed370dd012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" Nov 03 21:45:00.436728 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.436702 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/832c2242-e330-4541-961b-4acde0fbd38f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229\" (UID: \"832c2242-e330-4541-961b-4acde0fbd38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" Nov 03 21:45:00.436728 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.436719 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43eed4a-92c9-4074-936f-b48712db3dfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7\" (UID: \"f43eed4a-92c9-4074-936f-b48712db3dfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" Nov 03 21:45:00.436886 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.436754 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e997e7-9d8b-469b-ac1f-743fa96f7e1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895705-8ct6k\" (UID: \"64e997e7-9d8b-469b-ac1f-743fa96f7e1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" Nov 03 21:45:00.436886 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.436846 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4096e-7ea0-4377-a8a0-67c3c27d0036-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs\" (UID: \"c2b4096e-7ea0-4377-a8a0-67c3c27d0036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" Nov 03 21:45:00.436983 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.436966 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/947188bc-8b80-416c-b31a-c3ed370dd012-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl\" (UID: \"947188bc-8b80-416c-b31a-c3ed370dd012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" Nov 03 21:45:00.458443 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.458413 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d94ns\" (UniqueName: \"kubernetes.io/projected/832c2242-e330-4541-961b-4acde0fbd38f-kube-api-access-d94ns\") pod \"maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229\" (UID: \"832c2242-e330-4541-961b-4acde0fbd38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" Nov 03 21:45:00.458590 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.458519 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2rf8\" (UniqueName: \"kubernetes.io/projected/64e997e7-9d8b-469b-ac1f-743fa96f7e1f-kube-api-access-b2rf8\") pod \"maas-api-key-cleanup-29895705-8ct6k\" (UID: \"64e997e7-9d8b-469b-ac1f-743fa96f7e1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" Nov 03 21:45:00.458749 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.458726 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcp8n\" (UniqueName: \"kubernetes.io/projected/f43eed4a-92c9-4074-936f-b48712db3dfc-kube-api-access-bcp8n\") pod \"maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7\" (UID: \"f43eed4a-92c9-4074-936f-b48712db3dfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" Nov 03 21:45:00.458830 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.458809 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h24n9\" (UniqueName: \"kubernetes.io/projected/947188bc-8b80-416c-b31a-c3ed370dd012-kube-api-access-h24n9\") pod \"maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl\" (UID: \"947188bc-8b80-416c-b31a-c3ed370dd012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" Nov 03 21:45:00.458961 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.458946 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bd26\" (UniqueName: \"kubernetes.io/projected/c2b4096e-7ea0-4377-a8a0-67c3c27d0036-kube-api-access-4bd26\") pod \"maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs\" (UID: \"c2b4096e-7ea0-4377-a8a0-67c3c27d0036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" Nov 03 21:45:00.494689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.494635 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" Nov 03 21:45:00.502051 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.502030 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" Nov 03 21:45:00.564358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.563987 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" Nov 03 21:45:00.578247 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.578219 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" Nov 03 21:45:00.590097 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.589914 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" Nov 03 21:45:00.800846 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.800812 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs"] Nov 03 21:45:00.802151 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:45:00.802118 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b4096e_7ea0_4377_a8a0_67c3c27d0036.slice/crio-3baed2721adbbe24463d6665a0845e7f63499f5f2785d5f065f71bb25e21cc33 WatchSource:0}: Error finding container 3baed2721adbbe24463d6665a0845e7f63499f5f2785d5f065f71bb25e21cc33: Status 404 returned error can't find the container with id 3baed2721adbbe24463d6665a0845e7f63499f5f2785d5f065f71bb25e21cc33 Nov 03 21:45:00.816892 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.816860 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229"] Nov 03 21:45:00.905647 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:45:00.905614 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e997e7_9d8b_469b_ac1f_743fa96f7e1f.slice/crio-e272140713d6c8ed84fe42194961e6574987dd24a0aa6c4a73969ec0d9ffdf06 WatchSource:0}: Error finding container e272140713d6c8ed84fe42194961e6574987dd24a0aa6c4a73969ec0d9ffdf06: Status 404 returned error can't find the container with id e272140713d6c8ed84fe42194961e6574987dd24a0aa6c4a73969ec0d9ffdf06 Nov 03 21:45:00.906558 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.906539 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k"] Nov 03 21:45:00.938022 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:45:00.937993 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947188bc_8b80_416c_b31a_c3ed370dd012.slice/crio-e2173bcd8d4fcf5557312144b66b135fd016ed6d1e7a3cb4e5e1d02d77787a49 WatchSource:0}: Error finding container e2173bcd8d4fcf5557312144b66b135fd016ed6d1e7a3cb4e5e1d02d77787a49: Status 404 returned error can't find the container with id e2173bcd8d4fcf5557312144b66b135fd016ed6d1e7a3cb4e5e1d02d77787a49 Nov 03 21:45:00.938995 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:45:00.938963 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43eed4a_92c9_4074_936f_b48712db3dfc.slice/crio-b3649f5f4fe78849683806f4a6b079ab3d137e0b8bd3fdb167b210ea9f732e7c WatchSource:0}: Error finding container b3649f5f4fe78849683806f4a6b079ab3d137e0b8bd3fdb167b210ea9f732e7c: Status 404 returned error can't find the container with id b3649f5f4fe78849683806f4a6b079ab3d137e0b8bd3fdb167b210ea9f732e7c Nov 03 21:45:00.939681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.939641 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7"] Nov 03 21:45:00.946479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:00.946451 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl"] Nov 03 21:45:01.023514 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.023483 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2"] Nov 03 21:45:01.035534 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.035511 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.051351 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.051318 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2"] Nov 03 21:45:01.057902 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.057466 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-27fdf4\"/\"distinct-w2-27fdf4-kserve-self-signed-certs\"" Nov 03 21:45:01.152432 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.152394 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-home\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.152629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.152566 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-model-cache\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.152629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.152606 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r9bg\" (UniqueName: \"kubernetes.io/projected/afb7a890-c3ba-4d5f-bce1-713c58220d7d-kube-api-access-4r9bg\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.152784 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.152766 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-tmp-dir\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.152913 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.152895 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-dshm\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.152994 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.152974 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb7a890-c3ba-4d5f-bce1-713c58220d7d-tls-certs\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.256140 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.255618 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-model-cache\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.256791 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.256753 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r9bg\" (UniqueName: \"kubernetes.io/projected/afb7a890-c3ba-4d5f-bce1-713c58220d7d-kube-api-access-4r9bg\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.256961 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.256946 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-tmp-dir\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.257161 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.257145 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-dshm\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.257332 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.257317 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb7a890-c3ba-4d5f-bce1-713c58220d7d-tls-certs\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.257516 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.257503 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-home\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.258105 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.258062 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-home\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.259037 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.259017 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-tmp-dir\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.259505 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.259476 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-model-cache\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.267366 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.266309 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb7a890-c3ba-4d5f-bce1-713c58220d7d-tls-certs\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.270348 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.270328 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-dshm\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.296147 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.296100 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r9bg\" (UniqueName: \"kubernetes.io/projected/afb7a890-c3ba-4d5f-bce1-713c58220d7d-kube-api-access-4r9bg\") pod \"distinct-w2-27fdf4-kserve-554dd588d4-cdws2\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.364029 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.363994 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:01.675040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.675010 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2"] Nov 03 21:45:01.784615 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.784577 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" event={"ID":"afb7a890-c3ba-4d5f-bce1-713c58220d7d","Type":"ContainerStarted","Data":"faacd85db05e91963902cae054c4a1efac3cbc137a8e09d05c861188f4be4c10"} Nov 03 21:45:01.784812 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.784625 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" event={"ID":"afb7a890-c3ba-4d5f-bce1-713c58220d7d","Type":"ContainerStarted","Data":"281cdfcb57783dae89fd78b787eeaa8747b2502b892fe86427a428a25adb6141"} Nov 03 21:45:01.786824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.786793 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" event={"ID":"f43eed4a-92c9-4074-936f-b48712db3dfc","Type":"ContainerStarted","Data":"122e1fe59c0d557ba4268c9f743910e34994b02b75376a0c8f170374471fd8c8"} Nov 03 21:45:01.786964 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.786851 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" event={"ID":"f43eed4a-92c9-4074-936f-b48712db3dfc","Type":"ContainerStarted","Data":"b3649f5f4fe78849683806f4a6b079ab3d137e0b8bd3fdb167b210ea9f732e7c"} Nov 03 21:45:01.790216 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.790188 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" event={"ID":"64e997e7-9d8b-469b-ac1f-743fa96f7e1f","Type":"ContainerStarted","Data":"a237003fc4558c6353a894b8383c05f171a79ef8c46c63d9ea6f1b7c11b17a98"} Nov 03 21:45:01.790339 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.790237 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" event={"ID":"64e997e7-9d8b-469b-ac1f-743fa96f7e1f","Type":"ContainerStarted","Data":"e272140713d6c8ed84fe42194961e6574987dd24a0aa6c4a73969ec0d9ffdf06"} Nov 03 21:45:01.791837 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.791813 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" event={"ID":"c2b4096e-7ea0-4377-a8a0-67c3c27d0036","Type":"ContainerStarted","Data":"b72784402958c9b7f97f6c2253b30891695aa10ce29c7de324f5e31b15341b2f"} Nov 03 21:45:01.791946 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.791856 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" event={"ID":"c2b4096e-7ea0-4377-a8a0-67c3c27d0036","Type":"ContainerStarted","Data":"3baed2721adbbe24463d6665a0845e7f63499f5f2785d5f065f71bb25e21cc33"} Nov 03 21:45:01.794334 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.794312 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" event={"ID":"947188bc-8b80-416c-b31a-c3ed370dd012","Type":"ContainerStarted","Data":"5664a383d47d07470d047450bd3e779d7ba7e083324d63b4f6d569451c121316"} Nov 03 21:45:01.794431 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.794350 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" event={"ID":"947188bc-8b80-416c-b31a-c3ed370dd012","Type":"ContainerStarted","Data":"e2173bcd8d4fcf5557312144b66b135fd016ed6d1e7a3cb4e5e1d02d77787a49"} Nov 03 21:45:01.796606 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.796578 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" event={"ID":"832c2242-e330-4541-961b-4acde0fbd38f","Type":"ContainerStarted","Data":"6e200b8e76ee56266575975c9a38568e88d59d2c7645b97a8b193d96a090a7d4"} Nov 03 21:45:01.796727 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.796648 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" event={"ID":"832c2242-e330-4541-961b-4acde0fbd38f","Type":"ContainerStarted","Data":"84360bbc30709703c093d56546327539f1029c7fc9cfa6e4160965ebc51b76f1"} Nov 03 21:45:01.826593 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.826540 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podStartSLOduration=1.826526426 podStartE2EDuration="1.826526426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:01.824820753 +0000 UTC m=+917.332049034" watchObservedRunningTime="2026-11-03 21:45:01.826526426 +0000 UTC m=+917.333754705" Nov 03 21:45:01.865681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.865614 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" podStartSLOduration=1.865599631 podStartE2EDuration="1.865599631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:01.864035734 +0000 UTC m=+917.371264014" watchObservedRunningTime="2026-11-03 21:45:01.865599631 +0000 UTC m=+917.372827908" Nov 03 21:45:01.957110 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.957058 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" podStartSLOduration=1.957042653 podStartE2EDuration="1.957042653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:01.956346188 +0000 UTC m=+917.463574484" watchObservedRunningTime="2026-11-03 21:45:01.957042653 +0000 UTC m=+917.464270933" Nov 03 21:45:01.957826 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:01.957779 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" podStartSLOduration=1.957753493 podStartE2EDuration="1.957753493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:01.912853373 +0000 UTC m=+917.420081655" watchObservedRunningTime="2026-11-03 21:45:01.957753493 +0000 UTC m=+917.464981774" Nov 03 21:45:02.015091 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.014253 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" podStartSLOduration=2.014233865 podStartE2EDuration="2.014233865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:02.012352882 +0000 UTC m=+917.519581183" watchObservedRunningTime="2026-11-03 21:45:02.014233865 +0000 UTC m=+917.521462146" Nov 03 21:45:02.365149 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.365112 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:02.426780 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.426714 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" podStartSLOduration=2.426693724 podStartE2EDuration="2.426693724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:02.069280099 +0000 UTC m=+917.576508380" watchObservedRunningTime="2026-11-03 21:45:02.426693724 +0000 UTC m=+917.933922012" Nov 03 21:45:02.428267 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.428244 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9"] Nov 03 21:45:02.433767 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.433742 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" Nov 03 21:45:02.448805 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.448773 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9"] Nov 03 21:45:02.449511 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.449485 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:45:02.477895 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.477855 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq6sj\" (UniqueName: \"kubernetes.io/projected/b9256e44-c54f-4801-9cd7-bec18a050043-kube-api-access-lq6sj\") pod \"maas-api-revoke-sub-21af73a2-nwll9\" (UID: \"b9256e44-c54f-4801-9cd7-bec18a050043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" Nov 03 21:45:02.477895 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.477903 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9256e44-c54f-4801-9cd7-bec18a050043-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21af73a2-nwll9\" (UID: \"b9256e44-c54f-4801-9cd7-bec18a050043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" Nov 03 21:45:02.478111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.477993 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9256e44-c54f-4801-9cd7-bec18a050043-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21af73a2-nwll9\" (UID: \"b9256e44-c54f-4801-9cd7-bec18a050043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" Nov 03 21:45:02.579169 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.579131 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq6sj\" (UniqueName: \"kubernetes.io/projected/b9256e44-c54f-4801-9cd7-bec18a050043-kube-api-access-lq6sj\") pod \"maas-api-revoke-sub-21af73a2-nwll9\" (UID: \"b9256e44-c54f-4801-9cd7-bec18a050043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" Nov 03 21:45:02.579334 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.579177 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9256e44-c54f-4801-9cd7-bec18a050043-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21af73a2-nwll9\" (UID: \"b9256e44-c54f-4801-9cd7-bec18a050043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" Nov 03 21:45:02.579334 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.579243 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9256e44-c54f-4801-9cd7-bec18a050043-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21af73a2-nwll9\" (UID: \"b9256e44-c54f-4801-9cd7-bec18a050043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" Nov 03 21:45:02.580068 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.580037 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9256e44-c54f-4801-9cd7-bec18a050043-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21af73a2-nwll9\" (UID: \"b9256e44-c54f-4801-9cd7-bec18a050043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" Nov 03 21:45:02.581596 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.581574 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9256e44-c54f-4801-9cd7-bec18a050043-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21af73a2-nwll9\" (UID: \"b9256e44-c54f-4801-9cd7-bec18a050043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" Nov 03 21:45:02.600063 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.600036 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq6sj\" (UniqueName: \"kubernetes.io/projected/b9256e44-c54f-4801-9cd7-bec18a050043-kube-api-access-lq6sj\") pod \"maas-api-revoke-sub-21af73a2-nwll9\" (UID: \"b9256e44-c54f-4801-9cd7-bec18a050043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" Nov 03 21:45:02.751401 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.751367 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" Nov 03 21:45:02.814450 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.814360 5041 generic.go:356] "Generic (PLEG): container finished" podID="947188bc-8b80-416c-b31a-c3ed370dd012" containerID="5664a383d47d07470d047450bd3e779d7ba7e083324d63b4f6d569451c121316" exitCode=60 Nov 03 21:45:02.814561 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.814442 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" event={"ID":"947188bc-8b80-416c-b31a-c3ed370dd012","Type":"ContainerDied","Data":"5664a383d47d07470d047450bd3e779d7ba7e083324d63b4f6d569451c121316"} Nov 03 21:45:02.815206 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.814996 5041 scope.go:122] "RemoveContainer" containerID="5664a383d47d07470d047450bd3e779d7ba7e083324d63b4f6d569451c121316" Nov 03 21:45:02.829221 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.829190 5041 generic.go:356] "Generic (PLEG): container finished" podID="832c2242-e330-4541-961b-4acde0fbd38f" containerID="6e200b8e76ee56266575975c9a38568e88d59d2c7645b97a8b193d96a090a7d4" exitCode=60 Nov 03 21:45:02.829438 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.829414 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" event={"ID":"832c2242-e330-4541-961b-4acde0fbd38f","Type":"ContainerDied","Data":"6e200b8e76ee56266575975c9a38568e88d59d2c7645b97a8b193d96a090a7d4"} Nov 03 21:45:02.829924 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.829902 5041 scope.go:122] "RemoveContainer" containerID="6e200b8e76ee56266575975c9a38568e88d59d2c7645b97a8b193d96a090a7d4" Nov 03 21:45:02.834428 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.833893 5041 generic.go:356] "Generic (PLEG): container finished" podID="f43eed4a-92c9-4074-936f-b48712db3dfc" containerID="122e1fe59c0d557ba4268c9f743910e34994b02b75376a0c8f170374471fd8c8" exitCode=60 Nov 03 21:45:02.834428 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.834048 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" event={"ID":"f43eed4a-92c9-4074-936f-b48712db3dfc","Type":"ContainerDied","Data":"122e1fe59c0d557ba4268c9f743910e34994b02b75376a0c8f170374471fd8c8"} Nov 03 21:45:02.834428 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.834218 5041 scope.go:122] "RemoveContainer" containerID="122e1fe59c0d557ba4268c9f743910e34994b02b75376a0c8f170374471fd8c8" Nov 03 21:45:02.849250 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.849218 5041 generic.go:356] "Generic (PLEG): container finished" podID="64e997e7-9d8b-469b-ac1f-743fa96f7e1f" containerID="a237003fc4558c6353a894b8383c05f171a79ef8c46c63d9ea6f1b7c11b17a98" exitCode=60 Nov 03 21:45:02.849359 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.849309 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" event={"ID":"64e997e7-9d8b-469b-ac1f-743fa96f7e1f","Type":"ContainerDied","Data":"a237003fc4558c6353a894b8383c05f171a79ef8c46c63d9ea6f1b7c11b17a98"} Nov 03 21:45:02.849767 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.849744 5041 scope.go:122] "RemoveContainer" containerID="a237003fc4558c6353a894b8383c05f171a79ef8c46c63d9ea6f1b7c11b17a98" Nov 03 21:45:02.852431 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.852407 5041 generic.go:356] "Generic (PLEG): container finished" podID="c2b4096e-7ea0-4377-a8a0-67c3c27d0036" containerID="b72784402958c9b7f97f6c2253b30891695aa10ce29c7de324f5e31b15341b2f" exitCode=60 Nov 03 21:45:02.852795 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.852773 5041 scope.go:122] "RemoveContainer" containerID="b72784402958c9b7f97f6c2253b30891695aa10ce29c7de324f5e31b15341b2f" Nov 03 21:45:02.853974 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:02.853910 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" event={"ID":"c2b4096e-7ea0-4377-a8a0-67c3c27d0036","Type":"ContainerDied","Data":"b72784402958c9b7f97f6c2253b30891695aa10ce29c7de324f5e31b15341b2f"} Nov 03 21:45:03.063438 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:03.063410 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9"] Nov 03 21:45:03.872918 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:03.868732 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" event={"ID":"b9256e44-c54f-4801-9cd7-bec18a050043","Type":"ContainerStarted","Data":"d395c450f535d3c6d78fd035ea068dd829832fba790ebc6412c726559a5df867"} Nov 03 21:45:03.872918 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:03.868838 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" event={"ID":"b9256e44-c54f-4801-9cd7-bec18a050043","Type":"ContainerStarted","Data":"7036fdcb46eb4e6cb222412e74078d26a18d0569893f1791400800385655dc04"} Nov 03 21:45:03.872918 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:03.871360 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" event={"ID":"947188bc-8b80-416c-b31a-c3ed370dd012","Type":"ContainerStarted","Data":"0140387f00822be8b2019f30ada8de09987c4655c01c3da356d4653674a41f02"} Nov 03 21:45:03.873704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:03.873644 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" event={"ID":"832c2242-e330-4541-961b-4acde0fbd38f","Type":"ContainerStarted","Data":"95eae433ad398454ad5e3dacbf47efd55f473acf615d8d2ef773b70f49e961c8"} Nov 03 21:45:03.875719 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:03.875693 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" event={"ID":"f43eed4a-92c9-4074-936f-b48712db3dfc","Type":"ContainerStarted","Data":"0285468a542175cd101fb8e9d57312a466423b35460748b4f6c38f9053f138c0"} Nov 03 21:45:03.877672 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:03.877636 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" event={"ID":"64e997e7-9d8b-469b-ac1f-743fa96f7e1f","Type":"ContainerStarted","Data":"93faed90e08d10de2df7203b7a820d09826ac949c03b5b5a8a1b12689fe2f498"} Nov 03 21:45:03.879606 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:03.879587 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" event={"ID":"c2b4096e-7ea0-4377-a8a0-67c3c27d0036","Type":"ContainerStarted","Data":"ed01e64cf845bc27e967dbe53efe5896a322e2fc6c94819b4e59709629d4aa13"} Nov 03 21:45:03.914444 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:03.913926 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" podStartSLOduration=1.9139079730000002 podStartE2EDuration="1.913907973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:03.910511868 +0000 UTC m=+919.417740169" watchObservedRunningTime="2026-11-03 21:45:03.913907973 +0000 UTC m=+919.421136254" Nov 03 21:45:04.713669 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.713620 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv"] Nov 03 21:45:04.718145 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.718120 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.755210 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.755177 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv"] Nov 03 21:45:04.808844 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.808794 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277d0d55-608c-451a-a704-0a80454055bc-istio-podinfo\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.808844 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.808845 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7b9s\" (UniqueName: \"kubernetes.io/projected/277d0d55-608c-451a-a704-0a80454055bc-kube-api-access-d7b9s\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.809053 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.808976 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-istio-data\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.809102 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.809086 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277d0d55-608c-451a-a704-0a80454055bc-istio-token\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.809159 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.809144 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277d0d55-608c-451a-a704-0a80454055bc-istiod-ca-cert\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.809195 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.809182 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-workload-certs\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.809354 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.809338 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-credential-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.809401 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.809372 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-workload-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.809488 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.809466 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-istio-envoy\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.894449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.894411 5041 generic.go:356] "Generic (PLEG): container finished" podID="64e997e7-9d8b-469b-ac1f-743fa96f7e1f" containerID="93faed90e08d10de2df7203b7a820d09826ac949c03b5b5a8a1b12689fe2f498" exitCode=60 Nov 03 21:45:04.894923 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.894453 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" event={"ID":"64e997e7-9d8b-469b-ac1f-743fa96f7e1f","Type":"ContainerDied","Data":"93faed90e08d10de2df7203b7a820d09826ac949c03b5b5a8a1b12689fe2f498"} Nov 03 21:45:04.894923 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.894509 5041 scope.go:122] "RemoveContainer" containerID="a237003fc4558c6353a894b8383c05f171a79ef8c46c63d9ea6f1b7c11b17a98" Nov 03 21:45:04.894923 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.894824 5041 scope.go:122] "RemoveContainer" containerID="93faed90e08d10de2df7203b7a820d09826ac949c03b5b5a8a1b12689fe2f498" Nov 03 21:45:04.895193 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:45:04.895174 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895705-8ct6k_odh-ai-gateway-infra(64e997e7-9d8b-469b-ac1f-743fa96f7e1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" podUID="64e997e7-9d8b-469b-ac1f-743fa96f7e1f" Nov 03 21:45:04.896706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.896678 5041 generic.go:356] "Generic (PLEG): container finished" podID="c2b4096e-7ea0-4377-a8a0-67c3c27d0036" containerID="ed01e64cf845bc27e967dbe53efe5896a322e2fc6c94819b4e59709629d4aa13" exitCode=60 Nov 03 21:45:04.896832 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.896713 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" event={"ID":"c2b4096e-7ea0-4377-a8a0-67c3c27d0036","Type":"ContainerDied","Data":"ed01e64cf845bc27e967dbe53efe5896a322e2fc6c94819b4e59709629d4aa13"} Nov 03 21:45:04.896997 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.896980 5041 scope.go:122] "RemoveContainer" containerID="ed01e64cf845bc27e967dbe53efe5896a322e2fc6c94819b4e59709629d4aa13" Nov 03 21:45:04.897266 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:45:04.897248 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs_odh-ai-gateway-infra(c2b4096e-7ea0-4377-a8a0-67c3c27d0036)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" podUID="c2b4096e-7ea0-4377-a8a0-67c3c27d0036" Nov 03 21:45:04.898504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.898431 5041 generic.go:356] "Generic (PLEG): container finished" podID="b9256e44-c54f-4801-9cd7-bec18a050043" containerID="d395c450f535d3c6d78fd035ea068dd829832fba790ebc6412c726559a5df867" exitCode=0 Nov 03 21:45:04.898504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.898481 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" event={"ID":"b9256e44-c54f-4801-9cd7-bec18a050043","Type":"ContainerDied","Data":"d395c450f535d3c6d78fd035ea068dd829832fba790ebc6412c726559a5df867"} Nov 03 21:45:04.900611 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.900592 5041 generic.go:356] "Generic (PLEG): container finished" podID="947188bc-8b80-416c-b31a-c3ed370dd012" containerID="0140387f00822be8b2019f30ada8de09987c4655c01c3da356d4653674a41f02" exitCode=60 Nov 03 21:45:04.900734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.900686 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" event={"ID":"947188bc-8b80-416c-b31a-c3ed370dd012","Type":"ContainerDied","Data":"0140387f00822be8b2019f30ada8de09987c4655c01c3da356d4653674a41f02"} Nov 03 21:45:04.900945 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.900927 5041 scope.go:122] "RemoveContainer" containerID="0140387f00822be8b2019f30ada8de09987c4655c01c3da356d4653674a41f02" Nov 03 21:45:04.901251 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:45:04.901204 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl_odh-ai-gateway-infra(947188bc-8b80-416c-b31a-c3ed370dd012)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" podUID="947188bc-8b80-416c-b31a-c3ed370dd012" Nov 03 21:45:04.902859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.902807 5041 generic.go:356] "Generic (PLEG): container finished" podID="832c2242-e330-4541-961b-4acde0fbd38f" containerID="95eae433ad398454ad5e3dacbf47efd55f473acf615d8d2ef773b70f49e961c8" exitCode=60 Nov 03 21:45:04.902859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.902840 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" event={"ID":"832c2242-e330-4541-961b-4acde0fbd38f","Type":"ContainerDied","Data":"95eae433ad398454ad5e3dacbf47efd55f473acf615d8d2ef773b70f49e961c8"} Nov 03 21:45:04.903115 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.903094 5041 scope.go:122] "RemoveContainer" containerID="95eae433ad398454ad5e3dacbf47efd55f473acf615d8d2ef773b70f49e961c8" Nov 03 21:45:04.903431 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:45:04.903406 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229_odh-ai-gateway-infra(832c2242-e330-4541-961b-4acde0fbd38f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" podUID="832c2242-e330-4541-961b-4acde0fbd38f" Nov 03 21:45:04.904998 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.904976 5041 generic.go:356] "Generic (PLEG): container finished" podID="f43eed4a-92c9-4074-936f-b48712db3dfc" containerID="0285468a542175cd101fb8e9d57312a466423b35460748b4f6c38f9053f138c0" exitCode=60 Nov 03 21:45:04.905097 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.905003 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" event={"ID":"f43eed4a-92c9-4074-936f-b48712db3dfc","Type":"ContainerDied","Data":"0285468a542175cd101fb8e9d57312a466423b35460748b4f6c38f9053f138c0"} Nov 03 21:45:04.905264 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.905248 5041 scope.go:122] "RemoveContainer" containerID="0285468a542175cd101fb8e9d57312a466423b35460748b4f6c38f9053f138c0" Nov 03 21:45:04.905504 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:45:04.905485 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7_odh-ai-gateway-infra(f43eed4a-92c9-4074-936f-b48712db3dfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" podUID="f43eed4a-92c9-4074-936f-b48712db3dfc" Nov 03 21:45:04.910649 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.910615 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277d0d55-608c-451a-a704-0a80454055bc-istio-token\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.910780 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.910702 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277d0d55-608c-451a-a704-0a80454055bc-istiod-ca-cert\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.910780 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.910736 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-workload-certs\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.910887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.910842 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-credential-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.910887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.910870 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-workload-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.910993 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.910922 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-istio-envoy\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.911055 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.911008 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277d0d55-608c-451a-a704-0a80454055bc-istio-podinfo\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.911055 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.911032 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7b9s\" (UniqueName: \"kubernetes.io/projected/277d0d55-608c-451a-a704-0a80454055bc-kube-api-access-d7b9s\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.911156 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.911087 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-istio-data\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.911156 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.911132 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-workload-certs\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.911432 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.911401 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277d0d55-608c-451a-a704-0a80454055bc-istiod-ca-cert\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.911643 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.911621 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-workload-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.911791 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.911678 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-istio-data\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.911791 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.911780 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-credential-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.912480 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.912385 5041 scope.go:122] "RemoveContainer" containerID="b72784402958c9b7f97f6c2253b30891695aa10ce29c7de324f5e31b15341b2f" Nov 03 21:45:04.914208 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.914187 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277d0d55-608c-451a-a704-0a80454055bc-istio-podinfo\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.914508 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.914489 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-istio-envoy\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.938928 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.938890 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7b9s\" (UniqueName: \"kubernetes.io/projected/277d0d55-608c-451a-a704-0a80454055bc-kube-api-access-d7b9s\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.939083 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.938905 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277d0d55-608c-451a-a704-0a80454055bc-istio-token\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:04.971224 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:04.971183 5041 scope.go:122] "RemoveContainer" containerID="5664a383d47d07470d047450bd3e779d7ba7e083324d63b4f6d569451c121316" Nov 03 21:45:05.000680 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:05.000611 5041 scope.go:122] "RemoveContainer" containerID="6e200b8e76ee56266575975c9a38568e88d59d2c7645b97a8b193d96a090a7d4" Nov 03 21:45:05.016148 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:05.016121 5041 scope.go:122] "RemoveContainer" containerID="122e1fe59c0d557ba4268c9f743910e34994b02b75376a0c8f170374471fd8c8" Nov 03 21:45:05.032859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:05.032832 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:05.361903 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:05.361868 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv"] Nov 03 21:45:05.362217 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:45:05.362192 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277d0d55_608c_451a_a704_0a80454055bc.slice/crio-d675cc39093138ef1bdcf32bcc79e7113be7b474a44b48a700e48370a1434143 WatchSource:0}: Error finding container d675cc39093138ef1bdcf32bcc79e7113be7b474a44b48a700e48370a1434143: Status 404 returned error can't find the container with id d675cc39093138ef1bdcf32bcc79e7113be7b474a44b48a700e48370a1434143 Nov 03 21:45:05.365942 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:05.365905 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:05.366048 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:05.365989 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:05.366048 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:05.366030 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:05.918274 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:05.918235 5041 scope.go:122] "RemoveContainer" containerID="0285468a542175cd101fb8e9d57312a466423b35460748b4f6c38f9053f138c0" Nov 03 21:45:05.918744 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:45:05.918610 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7_odh-ai-gateway-infra(f43eed4a-92c9-4074-936f-b48712db3dfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" podUID="f43eed4a-92c9-4074-936f-b48712db3dfc" Nov 03 21:45:05.920372 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:05.920347 5041 scope.go:122] "RemoveContainer" containerID="93faed90e08d10de2df7203b7a820d09826ac949c03b5b5a8a1b12689fe2f498" Nov 03 21:45:05.920710 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:45:05.920655 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895705-8ct6k_odh-ai-gateway-infra(64e997e7-9d8b-469b-ac1f-743fa96f7e1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" podUID="64e997e7-9d8b-469b-ac1f-743fa96f7e1f" Nov 03 21:45:05.922552 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:05.922530 5041 scope.go:122] "RemoveContainer" containerID="ed01e64cf845bc27e967dbe53efe5896a322e2fc6c94819b4e59709629d4aa13" Nov 03 21:45:05.922890 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:45:05.922864 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs_odh-ai-gateway-infra(c2b4096e-7ea0-4377-a8a0-67c3c27d0036)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" podUID="c2b4096e-7ea0-4377-a8a0-67c3c27d0036" Nov 03 21:45:05.925479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:05.925459 5041 scope.go:122] "RemoveContainer" containerID="0140387f00822be8b2019f30ada8de09987c4655c01c3da356d4653674a41f02" Nov 03 21:45:05.925765 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:45:05.925742 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl_odh-ai-gateway-infra(947188bc-8b80-416c-b31a-c3ed370dd012)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" podUID="947188bc-8b80-416c-b31a-c3ed370dd012" Nov 03 21:45:05.927695 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:05.927638 5041 scope.go:122] "RemoveContainer" containerID="95eae433ad398454ad5e3dacbf47efd55f473acf615d8d2ef773b70f49e961c8" Nov 03 21:45:05.928099 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:45:05.927990 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229_odh-ai-gateway-infra(832c2242-e330-4541-961b-4acde0fbd38f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" podUID="832c2242-e330-4541-961b-4acde0fbd38f" Nov 03 21:45:05.929772 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:05.929749 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" event={"ID":"277d0d55-608c-451a-a704-0a80454055bc","Type":"ContainerStarted","Data":"bda53a79e99c30d00c517ce9375f3a9f4644f5469301dd2a173d3f70df0c83df"} Nov 03 21:45:05.929868 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:05.929800 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" event={"ID":"277d0d55-608c-451a-a704-0a80454055bc","Type":"ContainerStarted","Data":"d675cc39093138ef1bdcf32bcc79e7113be7b474a44b48a700e48370a1434143"} Nov 03 21:45:06.033549 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.033442 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:06.034548 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.034523 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 21:45:06.034752 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.034730 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" podUID="277d0d55-608c-451a-a704-0a80454055bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 21:45:06.099038 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.099015 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" Nov 03 21:45:06.128368 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.128336 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9256e44-c54f-4801-9cd7-bec18a050043-kube-api-access-lq6sj\" (UniqueName: \"kubernetes.io/projected/b9256e44-c54f-4801-9cd7-bec18a050043-kube-api-access-lq6sj\") pod \"b9256e44-c54f-4801-9cd7-bec18a050043\" (UID: \"b9256e44-c54f-4801-9cd7-bec18a050043\") " Nov 03 21:45:06.128977 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.128596 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9256e44-c54f-4801-9cd7-bec18a050043-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9256e44-c54f-4801-9cd7-bec18a050043-maas-api-xks-ca\") pod \"b9256e44-c54f-4801-9cd7-bec18a050043\" (UID: \"b9256e44-c54f-4801-9cd7-bec18a050043\") " Nov 03 21:45:06.128977 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.128731 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9256e44-c54f-4801-9cd7-bec18a050043-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9256e44-c54f-4801-9cd7-bec18a050043-maas-api-openshift-ca\") pod \"b9256e44-c54f-4801-9cd7-bec18a050043\" (UID: \"b9256e44-c54f-4801-9cd7-bec18a050043\") " Nov 03 21:45:06.129744 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.129380 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9256e44-c54f-4801-9cd7-bec18a050043-maas-api-openshift-ca" pod "b9256e44-c54f-4801-9cd7-bec18a050043" (UID: "b9256e44-c54f-4801-9cd7-bec18a050043"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:06.129744 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.129511 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9256e44-c54f-4801-9cd7-bec18a050043-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:06.132502 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.132477 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9256e44-c54f-4801-9cd7-bec18a050043-kube-api-access-lq6sj" pod "b9256e44-c54f-4801-9cd7-bec18a050043" (UID: "b9256e44-c54f-4801-9cd7-bec18a050043"). InnerVolumeSpecName "kube-api-access-lq6sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:06.133292 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.133266 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9256e44-c54f-4801-9cd7-bec18a050043-maas-api-xks-ca" pod "b9256e44-c54f-4801-9cd7-bec18a050043" (UID: "b9256e44-c54f-4801-9cd7-bec18a050043"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:45:06.133837 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.133783 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" podStartSLOduration=2.133767368 podStartE2EDuration="2.133767368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:06.127280555 +0000 UTC m=+921.634508873" watchObservedRunningTime="2026-11-03 21:45:06.133767368 +0000 UTC m=+921.640995649" Nov 03 21:45:06.238795 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.238718 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq6sj\" (UniqueName: \"kubernetes.io/projected/b9256e44-c54f-4801-9cd7-bec18a050043-kube-api-access-lq6sj\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:06.238795 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.238761 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9256e44-c54f-4801-9cd7-bec18a050043-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:06.986647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.986615 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" Nov 03 21:45:06.987587 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.987104 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9" event={"ID":"b9256e44-c54f-4801-9cd7-bec18a050043","Type":"ContainerDied","Data":"7036fdcb46eb4e6cb222412e74078d26a18d0569893f1791400800385655dc04"} Nov 03 21:45:06.987703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:06.987647 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7036fdcb46eb4e6cb222412e74078d26a18d0569893f1791400800385655dc04" Nov 03 21:45:07.039856 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:07.039820 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 21:45:07.040147 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:07.040117 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" podUID="277d0d55-608c-451a-a704-0a80454055bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 21:45:08.039143 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:08.039106 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 21:45:08.039701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:08.039168 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" podUID="277d0d55-608c-451a-a704-0a80454055bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 21:45:09.037055 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:09.037016 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 21:45:09.037350 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:09.037322 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" podUID="277d0d55-608c-451a-a704-0a80454055bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 21:45:09.799115 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:09.799071 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6"] Nov 03 21:45:09.800112 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:09.800082 5041 cpu_manager.go:414] "containerMap: removing container" podUID="b9256e44-c54f-4801-9cd7-bec18a050043" containerName="revoke-keys" Nov 03 21:45:09.800251 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:09.800113 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9256e44-c54f-4801-9cd7-bec18a050043" containerName="revoke-keys" Nov 03 21:45:09.800363 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:09.800347 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9256e44-c54f-4801-9cd7-bec18a050043" containerName="revoke-keys" Nov 03 21:45:09.806141 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:09.806118 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:09.884796 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:09.884751 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6"] Nov 03 21:45:10.109959 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.109858 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-envoy\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.130196 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.130160 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed859d9-82ee-46b1-9e96-776b7917656d-istiod-ca-cert\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.130510 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.130488 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-workload-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.130694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.130675 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-data\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.130857 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.130839 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-workload-certs\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.131115 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.131094 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-token\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.149876 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.149848 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-credential-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.150177 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.150158 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-podinfo\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.192686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.150348 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8s24\" (UniqueName: \"kubernetes.io/projected/6ed859d9-82ee-46b1-9e96-776b7917656d-kube-api-access-r8s24\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.262096 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.138438 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 21:45:10.262449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.262405 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" podUID="277d0d55-608c-451a-a704-0a80454055bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 21:45:10.337687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.331689 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2"] Nov 03 21:45:10.365766 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.345286 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.392236 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.392196 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-token\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.392579 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.392546 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-credential-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.392757 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.392738 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-podinfo\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.392926 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.392907 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8s24\" (UniqueName: \"kubernetes.io/projected/6ed859d9-82ee-46b1-9e96-776b7917656d-kube-api-access-r8s24\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.393065 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.393049 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-envoy\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.393200 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.393184 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed859d9-82ee-46b1-9e96-776b7917656d-istiod-ca-cert\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.393384 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.393368 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-workload-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.393538 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.393519 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-data\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.393676 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.393644 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-workload-certs\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.394413 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.394387 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-workload-certs\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.396250 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.396211 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-workload-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.397150 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.397123 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed859d9-82ee-46b1-9e96-776b7917656d-istiod-ca-cert\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.398352 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.397442 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-data\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.398352 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.397794 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-credential-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.398352 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.398004 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-envoy\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.401927 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.401886 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-podinfo\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.406830 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.406805 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2"] Nov 03 21:45:10.502735 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.496745 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8s24\" (UniqueName: \"kubernetes.io/projected/6ed859d9-82ee-46b1-9e96-776b7917656d-kube-api-access-r8s24\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.504947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.504528 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d87e1039-2359-4a69-a9ad-c029d9987a01-istiod-ca-cert\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.504947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.504772 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-data\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.504947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.504813 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-token\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.506227 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.505351 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-envoy\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.506227 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.505471 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-podinfo\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.506227 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.505496 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl657\" (UniqueName: \"kubernetes.io/projected/d87e1039-2359-4a69-a9ad-c029d9987a01-kube-api-access-tl657\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.506227 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.505548 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-credential-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.506227 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.505635 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-workload-certs\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.506227 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.505703 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-workload-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.519860 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.519817 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-token\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.586214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.586176 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd"] Nov 03 21:45:10.591305 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.591277 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.607392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.607356 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d87e1039-2359-4a69-a9ad-c029d9987a01-istiod-ca-cert\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.607614 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.607586 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-data\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.607757 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.607632 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-token\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.607757 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.607721 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-envoy\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.607893 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.607819 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-podinfo\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.607893 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.607841 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl657\" (UniqueName: \"kubernetes.io/projected/d87e1039-2359-4a69-a9ad-c029d9987a01-kube-api-access-tl657\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.607893 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.607887 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-credential-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.608052 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.607968 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-workload-certs\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.608052 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.608016 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-workload-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.630734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.608199 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d87e1039-2359-4a69-a9ad-c029d9987a01-istiod-ca-cert\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.649747 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.627991 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-workload-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.649747 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.628239 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-data\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.649978 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.647852 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-credential-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.649978 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.648903 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-podinfo\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.649978 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.649521 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-token\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.661400 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.661354 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-workload-certs\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.661751 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.661728 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd"] Nov 03 21:45:10.661992 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.661973 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-envoy\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.665828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.665808 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl657\" (UniqueName: \"kubernetes.io/projected/d87e1039-2359-4a69-a9ad-c029d9987a01-kube-api-access-tl657\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.714467 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.714426 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-podinfo\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.714849 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.714823 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-workload-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.715091 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.715072 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-token\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.735684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.734801 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-workload-certs\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.735684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.734879 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-credential-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.735684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.734922 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-envoy\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.735684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.735024 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7qh9\" (UniqueName: \"kubernetes.io/projected/d6132e6c-086e-47bc-9baa-4a341f85671c-kube-api-access-j7qh9\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.735684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.735107 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-data\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.735684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.735149 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6132e6c-086e-47bc-9baa-4a341f85671c-istiod-ca-cert\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.779222 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:45:10.779183 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:45:10.790357 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.790333 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:10.806581 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.806557 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:10.836899 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.836851 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-data\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.837039 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.836915 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6132e6c-086e-47bc-9baa-4a341f85671c-istiod-ca-cert\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.837039 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.836984 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-podinfo\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.837170 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.837073 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-workload-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.837170 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.837131 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-token\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.837276 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.837178 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-workload-certs\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.837276 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.837229 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-credential-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.837276 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.837265 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-envoy\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.837441 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.837351 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7qh9\" (UniqueName: \"kubernetes.io/projected/d6132e6c-086e-47bc-9baa-4a341f85671c-kube-api-access-j7qh9\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.838053 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.838017 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-data\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.838528 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.838508 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6132e6c-086e-47bc-9baa-4a341f85671c-istiod-ca-cert\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.839167 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.839113 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-workload-certs\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.839167 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.839142 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-workload-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.839321 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.839185 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-credential-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.841029 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.841006 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-envoy\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.841195 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.841173 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-podinfo\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.858878 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.858813 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7qh9\" (UniqueName: \"kubernetes.io/projected/d6132e6c-086e-47bc-9baa-4a341f85671c-kube-api-access-j7qh9\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.859055 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.859001 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-token\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:10.963921 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:10.963878 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:11.039684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.039093 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 21:45:11.039684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.039153 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" podUID="277d0d55-608c-451a-a704-0a80454055bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 21:45:11.168117 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.168082 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6"] Nov 03 21:45:11.173843 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.173054 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:11.173843 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.173134 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:11.173843 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.173172 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:11.187556 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.187519 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" event={"ID":"6ed859d9-82ee-46b1-9e96-776b7917656d","Type":"ContainerStarted","Data":"c4addd70e47f15bb9aa11aaff65f5aebcc2c478b4ba0f1757486e265656d556f"} Nov 03 21:45:11.190920 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.190857 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2"] Nov 03 21:45:11.192012 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:45:11.191988 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87e1039_2359_4a69_a9ad_c029d9987a01.slice/crio-c4c630265a8f9cccc190e92aa3562bc85d13fcc3616e6b5547f6e8ac496ae433 WatchSource:0}: Error finding container c4c630265a8f9cccc190e92aa3562bc85d13fcc3616e6b5547f6e8ac496ae433: Status 404 returned error can't find the container with id c4c630265a8f9cccc190e92aa3562bc85d13fcc3616e6b5547f6e8ac496ae433 Nov 03 21:45:11.196228 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.196193 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:11.196341 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.196275 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:11.196341 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.196321 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:11.359282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.352918 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd"] Nov 03 21:45:11.371354 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.370261 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:11.376008 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:45:11.375964 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6132e6c_086e_47bc_9baa_4a341f85671c.slice/crio-bf1d5883134a927b884aacc6dcb281ada25376461c0800c6caeb8674cc55cab0 WatchSource:0}: Error finding container bf1d5883134a927b884aacc6dcb281ada25376461c0800c6caeb8674cc55cab0: Status 404 returned error can't find the container with id bf1d5883134a927b884aacc6dcb281ada25376461c0800c6caeb8674cc55cab0 Nov 03 21:45:11.380961 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.379588 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:11.380961 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.379705 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:11.380961 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.379746 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:11.396961 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:11.396934 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:12.034328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.034281 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 21:45:12.034735 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.034352 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" podUID="277d0d55-608c-451a-a704-0a80454055bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 21:45:12.199973 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.199935 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" event={"ID":"d6132e6c-086e-47bc-9baa-4a341f85671c","Type":"ContainerStarted","Data":"320a8f899cc06d57d818f2b5f37146911aff2531e13003dcee0188e63603562e"} Nov 03 21:45:12.200142 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.199999 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" event={"ID":"d6132e6c-086e-47bc-9baa-4a341f85671c","Type":"ContainerStarted","Data":"bf1d5883134a927b884aacc6dcb281ada25376461c0800c6caeb8674cc55cab0"} Nov 03 21:45:12.202015 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.201981 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" event={"ID":"6ed859d9-82ee-46b1-9e96-776b7917656d","Type":"ContainerStarted","Data":"26b7768609b866b137d560a15c95e5b6978bb2181ac1affd75a2f3183cae6afa"} Nov 03 21:45:12.204792 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.204739 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" event={"ID":"d87e1039-2359-4a69-a9ad-c029d9987a01","Type":"ContainerStarted","Data":"7b580bc43fc5670c4a0773cef014bd9513a9c2bd657465507d6a9e7415127217"} Nov 03 21:45:12.205068 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.205046 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" event={"ID":"d87e1039-2359-4a69-a9ad-c029d9987a01","Type":"ContainerStarted","Data":"c4c630265a8f9cccc190e92aa3562bc85d13fcc3616e6b5547f6e8ac496ae433"} Nov 03 21:45:12.255624 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.255560 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" podStartSLOduration=2.25554049 podStartE2EDuration="2.25554049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:12.252648422 +0000 UTC m=+927.759876703" watchObservedRunningTime="2026-11-03 21:45:12.25554049 +0000 UTC m=+927.762768771" Nov 03 21:45:12.259888 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.259858 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:12.303770 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.303617 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" podStartSLOduration=3.303595132 podStartE2EDuration="3.303595132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:12.296455033 +0000 UTC m=+927.803683324" watchObservedRunningTime="2026-11-03 21:45:12.303595132 +0000 UTC m=+927.810823415" Nov 03 21:45:12.383610 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.383541 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" podStartSLOduration=3.383522631 podStartE2EDuration="3.383522631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:12.37616923 +0000 UTC m=+927.883397532" watchObservedRunningTime="2026-11-03 21:45:12.383522631 +0000 UTC m=+927.890750911" Nov 03 21:45:12.807701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.805944 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:12.807701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.806129 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 21:45:12.807701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.806175 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" podUID="6ed859d9-82ee-46b1-9e96-776b7917656d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 21:45:12.808761 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.808730 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:12.814853 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.814804 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:45:12.815115 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.815077 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" podUID="d87e1039-2359-4a69-a9ad-c029d9987a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 21:45:12.943763 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.943717 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62"] Nov 03 21:45:12.967201 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.967172 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:12.967493 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.967477 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" Nov 03 21:45:12.989684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.970768 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:45:12.989684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.970824 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" podUID="d6132e6c-086e-47bc-9baa-4a341f85671c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 21:45:12.989684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:12.988251 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:45:13.047689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.005394 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62"] Nov 03 21:45:13.113744 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.082968 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d838950-9fw62\" (UID: \"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" Nov 03 21:45:13.113744 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.083213 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc6v6\" (UniqueName: \"kubernetes.io/projected/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-kube-api-access-lc6v6\") pod \"maas-api-revoke-sub-6d838950-9fw62\" (UID: \"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" Nov 03 21:45:13.113744 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.083290 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d838950-9fw62\" (UID: \"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" Nov 03 21:45:13.113744 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.107050 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:13.197687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.187200 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:13.197687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.191772 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc6v6\" (UniqueName: \"kubernetes.io/projected/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-kube-api-access-lc6v6\") pod \"maas-api-revoke-sub-6d838950-9fw62\" (UID: \"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" Nov 03 21:45:13.197687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.191857 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d838950-9fw62\" (UID: \"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" Nov 03 21:45:13.197687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.191959 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d838950-9fw62\" (UID: \"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" Nov 03 21:45:13.222686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.215507 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d838950-9fw62\" (UID: \"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" Nov 03 21:45:13.400438 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.400360 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc6v6\" (UniqueName: \"kubernetes.io/projected/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-kube-api-access-lc6v6\") pod \"maas-api-revoke-sub-6d838950-9fw62\" (UID: \"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" Nov 03 21:45:13.434688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.417755 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:45:13.489688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.486021 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d838950-9fw62\" (UID: \"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" Nov 03 21:45:13.489688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.486337 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" Nov 03 21:45:13.818687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.817599 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 21:45:13.818687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.817681 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" podUID="6ed859d9-82ee-46b1-9e96-776b7917656d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 21:45:13.858683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.855797 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:45:13.858683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.855863 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" podUID="d87e1039-2359-4a69-a9ad-c029d9987a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 21:45:13.975698 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.974242 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:45:13.975698 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:13.974307 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" podUID="d6132e6c-086e-47bc-9baa-4a341f85671c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 21:45:14.605364 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:14.603443 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:45:14.676211 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:14.675479 5041 scope.go:122] "RemoveContainer" containerID="95eae433ad398454ad5e3dacbf47efd55f473acf615d8d2ef773b70f49e961c8" Nov 03 21:45:14.871272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:14.869910 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:45:14.871272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:14.869998 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" podUID="d87e1039-2359-4a69-a9ad-c029d9987a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 21:45:14.904738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:14.876423 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 21:45:14.904738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:14.876490 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" podUID="6ed859d9-82ee-46b1-9e96-776b7917656d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 21:45:15.039417 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:15.038827 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:45:15.039417 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:15.038892 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" podUID="d6132e6c-086e-47bc-9baa-4a341f85671c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 21:45:15.176933 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:15.149985 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:45:15.176933 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:15.176051 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62"] Nov 03 21:45:15.793816 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:15.791216 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 21:45:15.793816 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:15.791298 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" podUID="6ed859d9-82ee-46b1-9e96-776b7917656d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 21:45:15.810064 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:15.810017 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:45:15.810333 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:15.810305 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" podUID="d87e1039-2359-4a69-a9ad-c029d9987a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 21:45:16.020685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:16.002643 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:45:16.102774 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:16.025734 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" podUID="d6132e6c-086e-47bc-9baa-4a341f85671c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 21:45:16.234967 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:16.227899 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 21:45:16.373889 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:16.373783 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:45:16.481769 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:16.481726 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" event={"ID":"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19","Type":"ContainerStarted","Data":"4d3fb494c2ca08a5515d62a8305479a33020392b14e086747c11e48ae8bafac6"} Nov 03 21:45:16.484727 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:16.484698 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" event={"ID":"832c2242-e330-4541-961b-4acde0fbd38f","Type":"ContainerStarted","Data":"3725da050571a586424315c7b2af364eb1b574b338e6a18a1d024f627da2d8dc"} Nov 03 21:45:16.651751 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:16.632741 5041 scope.go:122] "RemoveContainer" containerID="ed01e64cf845bc27e967dbe53efe5896a322e2fc6c94819b4e59709629d4aa13" Nov 03 21:45:16.652591 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:16.652550 5041 scope.go:122] "RemoveContainer" containerID="0140387f00822be8b2019f30ada8de09987c4655c01c3da356d4653674a41f02" Nov 03 21:45:16.660448 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:16.660419 5041 scope.go:122] "RemoveContainer" containerID="93faed90e08d10de2df7203b7a820d09826ac949c03b5b5a8a1b12689fe2f498" Nov 03 21:45:16.861923 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:16.831409 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 21:45:17.155444 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:17.155399 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" podUID="6ed859d9-82ee-46b1-9e96-776b7917656d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 21:45:17.156206 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:16.832372 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:45:17.156422 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:17.156387 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" podUID="d87e1039-2359-4a69-a9ad-c029d9987a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 21:45:17.156563 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:17.093373 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:45:17.156716 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:17.156695 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" podUID="d6132e6c-086e-47bc-9baa-4a341f85671c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 21:45:17.364894 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:17.362768 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:45:17.364894 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:17.363395 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:17.382176 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:17.382126 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:45:17.389801 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:17.389753 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:45:17.724688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:17.698593 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:45:17.788760 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:17.788715 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:45:17.790994 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:17.790967 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 21:45:17.791169 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:17.791144 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" podUID="6ed859d9-82ee-46b1-9e96-776b7917656d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 21:45:17.858746 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:17.858027 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229"] Nov 03 21:45:17.891290 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:17.891247 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:45:17.891892 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:17.891842 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" podUID="d87e1039-2359-4a69-a9ad-c029d9987a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 21:45:18.083733 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:18.078653 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:45:18.113780 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:18.107923 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:45:18.113780 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:18.107990 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" podUID="d6132e6c-086e-47bc-9baa-4a341f85671c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 21:45:18.262688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:18.237899 5041 generic.go:356] "Generic (PLEG): container finished" podID="832c2242-e330-4541-961b-4acde0fbd38f" containerID="3725da050571a586424315c7b2af364eb1b574b338e6a18a1d024f627da2d8dc" exitCode=60 Nov 03 21:45:18.288014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:18.261181 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" event={"ID":"832c2242-e330-4541-961b-4acde0fbd38f","Type":"ContainerDied","Data":"3725da050571a586424315c7b2af364eb1b574b338e6a18a1d024f627da2d8dc"} Nov 03 21:45:18.292690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:18.292459 5041 scope.go:122] "RemoveContainer" containerID="95eae433ad398454ad5e3dacbf47efd55f473acf615d8d2ef773b70f49e961c8" Nov 03 21:45:18.300380 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:18.300336 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" event={"ID":"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19","Type":"ContainerStarted","Data":"deb16faa31afc3dc89797176f21ace5376dce98c22f471a4546c43e96f0b42fa"} Nov 03 21:45:18.305934 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:18.305911 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:18.665208 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:18.665179 5041 scope.go:122] "RemoveContainer" containerID="0285468a542175cd101fb8e9d57312a466423b35460748b4f6c38f9053f138c0" Nov 03 21:45:18.810724 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:18.807874 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 21:45:18.810724 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:18.807950 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" podUID="6ed859d9-82ee-46b1-9e96-776b7917656d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 21:45:18.810724 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:18.808873 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 21:45:18.810724 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:18.808916 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" podUID="d87e1039-2359-4a69-a9ad-c029d9987a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 21:45:18.895384 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:18.895355 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" Nov 03 21:45:18.973696 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:18.969027 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 21:45:18.973696 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:18.969091 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" podUID="d6132e6c-086e-47bc-9baa-4a341f85671c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 21:45:19.015681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.000585 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832c2242-e330-4541-961b-4acde0fbd38f-kube-api-access-d94ns\" (UniqueName: \"kubernetes.io/projected/832c2242-e330-4541-961b-4acde0fbd38f-kube-api-access-d94ns\") pod \"832c2242-e330-4541-961b-4acde0fbd38f\" (UID: \"832c2242-e330-4541-961b-4acde0fbd38f\") " Nov 03 21:45:19.015681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.000862 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/832c2242-e330-4541-961b-4acde0fbd38f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/832c2242-e330-4541-961b-4acde0fbd38f-maas-api-service-ca\") pod \"832c2242-e330-4541-961b-4acde0fbd38f\" (UID: \"832c2242-e330-4541-961b-4acde0fbd38f\") " Nov 03 21:45:19.015681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.008938 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/832c2242-e330-4541-961b-4acde0fbd38f-maas-api-service-ca" pod "832c2242-e330-4541-961b-4acde0fbd38f" (UID: "832c2242-e330-4541-961b-4acde0fbd38f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:19.045979 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.045941 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832c2242-e330-4541-961b-4acde0fbd38f-kube-api-access-d94ns" pod "832c2242-e330-4541-961b-4acde0fbd38f" (UID: "832c2242-e330-4541-961b-4acde0fbd38f"). InnerVolumeSpecName "kube-api-access-d94ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:19.103502 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.103455 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/832c2242-e330-4541-961b-4acde0fbd38f-maas-api-service-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:19.103502 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.103510 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d94ns\" (UniqueName: \"kubernetes.io/projected/832c2242-e330-4541-961b-4acde0fbd38f-kube-api-access-d94ns\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:19.335701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.329420 5041 generic.go:356] "Generic (PLEG): container finished" podID="64e997e7-9d8b-469b-ac1f-743fa96f7e1f" containerID="25ca0460bb219959e5ce2f6081c9d29bc4c2e079dd1c5d80947c25681612267c" exitCode=60 Nov 03 21:45:19.335701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.329505 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" event={"ID":"64e997e7-9d8b-469b-ac1f-743fa96f7e1f","Type":"ContainerDied","Data":"25ca0460bb219959e5ce2f6081c9d29bc4c2e079dd1c5d80947c25681612267c"} Nov 03 21:45:19.335701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.329597 5041 scope.go:122] "RemoveContainer" containerID="93faed90e08d10de2df7203b7a820d09826ac949c03b5b5a8a1b12689fe2f498" Nov 03 21:45:19.335701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.330089 5041 scope.go:122] "RemoveContainer" containerID="25ca0460bb219959e5ce2f6081c9d29bc4c2e079dd1c5d80947c25681612267c" Nov 03 21:45:19.335701 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:45:19.330465 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895705-8ct6k_odh-ai-gateway-infra(64e997e7-9d8b-469b-ac1f-743fa96f7e1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" podUID="64e997e7-9d8b-469b-ac1f-743fa96f7e1f" Nov 03 21:45:19.340240 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.338878 5041 generic.go:356] "Generic (PLEG): container finished" podID="c2b4096e-7ea0-4377-a8a0-67c3c27d0036" containerID="7727916f85b006da355e378b8e3f17561402d0dbade80268d909f2e57f038d5a" exitCode=60 Nov 03 21:45:19.340240 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.339070 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" event={"ID":"c2b4096e-7ea0-4377-a8a0-67c3c27d0036","Type":"ContainerDied","Data":"7727916f85b006da355e378b8e3f17561402d0dbade80268d909f2e57f038d5a"} Nov 03 21:45:19.340240 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.339502 5041 scope.go:122] "RemoveContainer" containerID="7727916f85b006da355e378b8e3f17561402d0dbade80268d909f2e57f038d5a" Nov 03 21:45:19.343005 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.342473 5041 generic.go:356] "Generic (PLEG): container finished" podID="947188bc-8b80-416c-b31a-c3ed370dd012" containerID="a601e09f6a15dc97af0d5ff65d132e5b5ea714097748dd1b66c8eb9a37723028" exitCode=60 Nov 03 21:45:19.343005 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.342551 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" event={"ID":"947188bc-8b80-416c-b31a-c3ed370dd012","Type":"ContainerDied","Data":"a601e09f6a15dc97af0d5ff65d132e5b5ea714097748dd1b66c8eb9a37723028"} Nov 03 21:45:19.369487 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.366462 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" event={"ID":"832c2242-e330-4541-961b-4acde0fbd38f","Type":"ContainerDied","Data":"84360bbc30709703c093d56546327539f1029c7fc9cfa6e4160965ebc51b76f1"} Nov 03 21:45:19.369487 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.366594 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229" Nov 03 21:45:19.392696 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:45:19.386842 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs_odh-ai-gateway-infra(c2b4096e-7ea0-4377-a8a0-67c3c27d0036)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" podUID="c2b4096e-7ea0-4377-a8a0-67c3c27d0036" Nov 03 21:45:19.392696 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.387285 5041 scope.go:122] "RemoveContainer" containerID="a601e09f6a15dc97af0d5ff65d132e5b5ea714097748dd1b66c8eb9a37723028" Nov 03 21:45:19.392696 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:45:19.387585 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl_odh-ai-gateway-infra(947188bc-8b80-416c-b31a-c3ed370dd012)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" podUID="947188bc-8b80-416c-b31a-c3ed370dd012" Nov 03 21:45:19.412859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.412829 5041 scope.go:122] "RemoveContainer" containerID="ed01e64cf845bc27e967dbe53efe5896a322e2fc6c94819b4e59709629d4aa13" Nov 03 21:45:19.413903 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.413874 5041 generic.go:356] "Generic (PLEG): container finished" podID="733c68bd-9bf7-46ee-8b4f-b1aa3144fa19" containerID="deb16faa31afc3dc89797176f21ace5376dce98c22f471a4546c43e96f0b42fa" exitCode=0 Nov 03 21:45:19.414026 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.413998 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" event={"ID":"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19","Type":"ContainerDied","Data":"deb16faa31afc3dc89797176f21ace5376dce98c22f471a4546c43e96f0b42fa"} Nov 03 21:45:19.420460 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.417877 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" event={"ID":"f43eed4a-92c9-4074-936f-b48712db3dfc","Type":"ContainerStarted","Data":"8ff5fa6ac19a0764f4aa573547945f50bcfd0125910e86d17a21d13fb2e5a9c2"} Nov 03 21:45:19.455612 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.455581 5041 scope.go:122] "RemoveContainer" containerID="0140387f00822be8b2019f30ada8de09987c4655c01c3da356d4653674a41f02" Nov 03 21:45:19.473113 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.473095 5041 scope.go:122] "RemoveContainer" containerID="3725da050571a586424315c7b2af364eb1b574b338e6a18a1d024f627da2d8dc" Nov 03 21:45:19.578113 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.578083 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229"] Nov 03 21:45:19.585810 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.585774 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a6da1-29895705-4t229"] Nov 03 21:45:19.670609 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.670575 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" Nov 03 21:45:19.794744 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.794230 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-maas-api-openshift-ca\") pod \"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19\" (UID: \"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19\") " Nov 03 21:45:19.794744 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.794371 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-maas-api-xks-ca\") pod \"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19\" (UID: \"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19\") " Nov 03 21:45:19.794744 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.794556 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-kube-api-access-lc6v6\" (UniqueName: \"kubernetes.io/projected/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-kube-api-access-lc6v6\") pod \"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19\" (UID: \"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19\") " Nov 03 21:45:19.802739 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.795887 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-maas-api-openshift-ca" pod "733c68bd-9bf7-46ee-8b4f-b1aa3144fa19" (UID: "733c68bd-9bf7-46ee-8b4f-b1aa3144fa19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:19.802739 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.798692 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-maas-api-xks-ca" pod "733c68bd-9bf7-46ee-8b4f-b1aa3144fa19" (UID: "733c68bd-9bf7-46ee-8b4f-b1aa3144fa19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:45:19.803963 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.803919 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-kube-api-access-lc6v6" pod "733c68bd-9bf7-46ee-8b4f-b1aa3144fa19" (UID: "733c68bd-9bf7-46ee-8b4f-b1aa3144fa19"). InnerVolumeSpecName "kube-api-access-lc6v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:19.820709 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.814393 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:19.820709 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.816915 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:19.820709 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.818022 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:45:19.897740 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.897708 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:19.898248 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.897745 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc6v6\" (UniqueName: \"kubernetes.io/projected/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-kube-api-access-lc6v6\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:19.898248 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:19.897763 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:20.179292 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.179213 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:20.179723 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.179701 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:20.181106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.181084 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:45:20.255966 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.255921 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:20.256477 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.256423 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:20.258223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.258199 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:45:20.409675 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.409640 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k"] Nov 03 21:45:20.445577 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.445543 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl"] Nov 03 21:45:20.449508 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.449482 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" Nov 03 21:45:20.449635 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.449522 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62" event={"ID":"733c68bd-9bf7-46ee-8b4f-b1aa3144fa19","Type":"ContainerDied","Data":"4d3fb494c2ca08a5515d62a8305479a33020392b14e086747c11e48ae8bafac6"} Nov 03 21:45:20.449635 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.449577 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d3fb494c2ca08a5515d62a8305479a33020392b14e086747c11e48ae8bafac6" Nov 03 21:45:20.454211 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.454182 5041 generic.go:356] "Generic (PLEG): container finished" podID="f43eed4a-92c9-4074-936f-b48712db3dfc" containerID="8ff5fa6ac19a0764f4aa573547945f50bcfd0125910e86d17a21d13fb2e5a9c2" exitCode=60 Nov 03 21:45:20.455265 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.455236 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" event={"ID":"f43eed4a-92c9-4074-936f-b48712db3dfc","Type":"ContainerDied","Data":"8ff5fa6ac19a0764f4aa573547945f50bcfd0125910e86d17a21d13fb2e5a9c2"} Nov 03 21:45:20.455395 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.455301 5041 scope.go:122] "RemoveContainer" containerID="0285468a542175cd101fb8e9d57312a466423b35460748b4f6c38f9053f138c0" Nov 03 21:45:20.456087 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.456058 5041 scope.go:122] "RemoveContainer" containerID="8ff5fa6ac19a0764f4aa573547945f50bcfd0125910e86d17a21d13fb2e5a9c2" Nov 03 21:45:20.456400 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:45:20.456374 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7_odh-ai-gateway-infra(f43eed4a-92c9-4074-936f-b48712db3dfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" podUID="f43eed4a-92c9-4074-936f-b48712db3dfc" Nov 03 21:45:20.502001 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.501634 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs"] Nov 03 21:45:20.554737 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.554459 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7"] Nov 03 21:45:20.635515 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.635447 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="832c2242-e330-4541-961b-4acde0fbd38f" path="/var/lib/kubelet/pods/832c2242-e330-4541-961b-4acde0fbd38f/volumes" Nov 03 21:45:20.689525 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.689481 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" Nov 03 21:45:20.767825 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.767795 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" Nov 03 21:45:20.797031 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.797000 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" Nov 03 21:45:20.822739 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.822710 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/947188bc-8b80-416c-b31a-c3ed370dd012-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/947188bc-8b80-416c-b31a-c3ed370dd012-maas-api-service-ca\") pod \"947188bc-8b80-416c-b31a-c3ed370dd012\" (UID: \"947188bc-8b80-416c-b31a-c3ed370dd012\") " Nov 03 21:45:20.823017 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.822999 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e997e7-9d8b-469b-ac1f-743fa96f7e1f-kube-api-access-b2rf8\" (UniqueName: \"kubernetes.io/projected/64e997e7-9d8b-469b-ac1f-743fa96f7e1f-kube-api-access-b2rf8\") pod \"64e997e7-9d8b-469b-ac1f-743fa96f7e1f\" (UID: \"64e997e7-9d8b-469b-ac1f-743fa96f7e1f\") " Nov 03 21:45:20.823149 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.823135 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e997e7-9d8b-469b-ac1f-743fa96f7e1f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e997e7-9d8b-469b-ac1f-743fa96f7e1f-maas-api-service-ca\") pod \"64e997e7-9d8b-469b-ac1f-743fa96f7e1f\" (UID: \"64e997e7-9d8b-469b-ac1f-743fa96f7e1f\") " Nov 03 21:45:20.823304 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.823290 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/947188bc-8b80-416c-b31a-c3ed370dd012-kube-api-access-h24n9\" (UniqueName: \"kubernetes.io/projected/947188bc-8b80-416c-b31a-c3ed370dd012-kube-api-access-h24n9\") pod \"947188bc-8b80-416c-b31a-c3ed370dd012\" (UID: \"947188bc-8b80-416c-b31a-c3ed370dd012\") " Nov 03 21:45:20.823791 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.823133 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947188bc-8b80-416c-b31a-c3ed370dd012-maas-api-service-ca" pod "947188bc-8b80-416c-b31a-c3ed370dd012" (UID: "947188bc-8b80-416c-b31a-c3ed370dd012"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:20.823900 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.823755 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e997e7-9d8b-469b-ac1f-743fa96f7e1f-maas-api-service-ca" pod "64e997e7-9d8b-469b-ac1f-743fa96f7e1f" (UID: "64e997e7-9d8b-469b-ac1f-743fa96f7e1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:20.824192 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.824169 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/947188bc-8b80-416c-b31a-c3ed370dd012-maas-api-service-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:20.824313 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.824296 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e997e7-9d8b-469b-ac1f-743fa96f7e1f-maas-api-service-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:20.832994 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.832965 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947188bc-8b80-416c-b31a-c3ed370dd012-kube-api-access-h24n9" pod "947188bc-8b80-416c-b31a-c3ed370dd012" (UID: "947188bc-8b80-416c-b31a-c3ed370dd012"). InnerVolumeSpecName "kube-api-access-h24n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:20.834850 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.834809 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e997e7-9d8b-469b-ac1f-743fa96f7e1f-kube-api-access-b2rf8" pod "64e997e7-9d8b-469b-ac1f-743fa96f7e1f" (UID: "64e997e7-9d8b-469b-ac1f-743fa96f7e1f"). InnerVolumeSpecName "kube-api-access-b2rf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:20.925219 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.925188 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b4096e-7ea0-4377-a8a0-67c3c27d0036-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4096e-7ea0-4377-a8a0-67c3c27d0036-maas-api-service-ca\") pod \"c2b4096e-7ea0-4377-a8a0-67c3c27d0036\" (UID: \"c2b4096e-7ea0-4377-a8a0-67c3c27d0036\") " Nov 03 21:45:20.925586 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.925237 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b4096e-7ea0-4377-a8a0-67c3c27d0036-kube-api-access-4bd26\" (UniqueName: \"kubernetes.io/projected/c2b4096e-7ea0-4377-a8a0-67c3c27d0036-kube-api-access-4bd26\") pod \"c2b4096e-7ea0-4377-a8a0-67c3c27d0036\" (UID: \"c2b4096e-7ea0-4377-a8a0-67c3c27d0036\") " Nov 03 21:45:20.925586 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.925518 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b4096e-7ea0-4377-a8a0-67c3c27d0036-maas-api-service-ca" pod "c2b4096e-7ea0-4377-a8a0-67c3c27d0036" (UID: "c2b4096e-7ea0-4377-a8a0-67c3c27d0036"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:20.925768 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.925753 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2rf8\" (UniqueName: \"kubernetes.io/projected/64e997e7-9d8b-469b-ac1f-743fa96f7e1f-kube-api-access-b2rf8\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:20.925823 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.925769 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4096e-7ea0-4377-a8a0-67c3c27d0036-maas-api-service-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:20.925823 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.925779 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h24n9\" (UniqueName: \"kubernetes.io/projected/947188bc-8b80-416c-b31a-c3ed370dd012-kube-api-access-h24n9\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:20.927363 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:20.927338 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b4096e-7ea0-4377-a8a0-67c3c27d0036-kube-api-access-4bd26" pod "c2b4096e-7ea0-4377-a8a0-67c3c27d0036" (UID: "c2b4096e-7ea0-4377-a8a0-67c3c27d0036"). InnerVolumeSpecName "kube-api-access-4bd26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:21.026732 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.026694 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bd26\" (UniqueName: \"kubernetes.io/projected/c2b4096e-7ea0-4377-a8a0-67c3c27d0036-kube-api-access-4bd26\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:21.278475 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.278392 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs"] Nov 03 21:45:21.279068 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279052 5041 cpu_manager.go:414] "containerMap: removing container" podUID="832c2242-e330-4541-961b-4acde0fbd38f" containerName="cleanup" Nov 03 21:45:21.279068 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279068 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832c2242-e330-4541-961b-4acde0fbd38f" containerName="cleanup" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279077 5041 cpu_manager.go:414] "containerMap: removing container" podUID="c2b4096e-7ea0-4377-a8a0-67c3c27d0036" containerName="cleanup" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279082 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b4096e-7ea0-4377-a8a0-67c3c27d0036" containerName="cleanup" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279091 5041 cpu_manager.go:414] "containerMap: removing container" podUID="832c2242-e330-4541-961b-4acde0fbd38f" containerName="cleanup" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279096 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832c2242-e330-4541-961b-4acde0fbd38f" containerName="cleanup" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279108 5041 cpu_manager.go:414] "containerMap: removing container" podUID="947188bc-8b80-416c-b31a-c3ed370dd012" containerName="cleanup" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279113 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947188bc-8b80-416c-b31a-c3ed370dd012" containerName="cleanup" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279127 5041 cpu_manager.go:414] "containerMap: removing container" podUID="c2b4096e-7ea0-4377-a8a0-67c3c27d0036" containerName="cleanup" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279131 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b4096e-7ea0-4377-a8a0-67c3c27d0036" containerName="cleanup" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279137 5041 cpu_manager.go:414] "containerMap: removing container" podUID="64e997e7-9d8b-469b-ac1f-743fa96f7e1f" containerName="cleanup" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279142 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e997e7-9d8b-469b-ac1f-743fa96f7e1f" containerName="cleanup" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279156 5041 cpu_manager.go:414] "containerMap: removing container" podUID="c2b4096e-7ea0-4377-a8a0-67c3c27d0036" containerName="cleanup" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279160 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b4096e-7ea0-4377-a8a0-67c3c27d0036" containerName="cleanup" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279175 5041 cpu_manager.go:414] "containerMap: removing container" podUID="733c68bd-9bf7-46ee-8b4f-b1aa3144fa19" containerName="revoke-keys" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279180 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733c68bd-9bf7-46ee-8b4f-b1aa3144fa19" containerName="revoke-keys" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279197 5041 cpu_manager.go:414] "containerMap: removing container" podUID="947188bc-8b80-416c-b31a-c3ed370dd012" containerName="cleanup" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279204 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947188bc-8b80-416c-b31a-c3ed370dd012" containerName="cleanup" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279213 5041 cpu_manager.go:414] "containerMap: removing container" podUID="947188bc-8b80-416c-b31a-c3ed370dd012" containerName="cleanup" Nov 03 21:45:21.279214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279219 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947188bc-8b80-416c-b31a-c3ed370dd012" containerName="cleanup" Nov 03 21:45:21.280045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279230 5041 cpu_manager.go:414] "containerMap: removing container" podUID="64e997e7-9d8b-469b-ac1f-743fa96f7e1f" containerName="cleanup" Nov 03 21:45:21.280045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279237 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e997e7-9d8b-469b-ac1f-743fa96f7e1f" containerName="cleanup" Nov 03 21:45:21.280045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279375 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="832c2242-e330-4541-961b-4acde0fbd38f" containerName="cleanup" Nov 03 21:45:21.280045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279385 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b4096e-7ea0-4377-a8a0-67c3c27d0036" containerName="cleanup" Nov 03 21:45:21.280045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279392 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="832c2242-e330-4541-961b-4acde0fbd38f" containerName="cleanup" Nov 03 21:45:21.280045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279400 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="947188bc-8b80-416c-b31a-c3ed370dd012" containerName="cleanup" Nov 03 21:45:21.280045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279407 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b4096e-7ea0-4377-a8a0-67c3c27d0036" containerName="cleanup" Nov 03 21:45:21.280045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279414 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e997e7-9d8b-469b-ac1f-743fa96f7e1f" containerName="cleanup" Nov 03 21:45:21.280045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279422 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b4096e-7ea0-4377-a8a0-67c3c27d0036" containerName="cleanup" Nov 03 21:45:21.280045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279431 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="733c68bd-9bf7-46ee-8b4f-b1aa3144fa19" containerName="revoke-keys" Nov 03 21:45:21.280045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279443 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e997e7-9d8b-469b-ac1f-743fa96f7e1f" containerName="cleanup" Nov 03 21:45:21.280045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279455 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="947188bc-8b80-416c-b31a-c3ed370dd012" containerName="cleanup" Nov 03 21:45:21.280045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.279461 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="947188bc-8b80-416c-b31a-c3ed370dd012" containerName="cleanup" Nov 03 21:45:21.283402 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.283384 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.304376 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.304346 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd5620\"/\"distinct-2-w3-dd5620-kserve-self-signed-certs\"" Nov 03 21:45:21.306576 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.306547 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs"] Nov 03 21:45:21.438303 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.438265 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-model-cache\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.438303 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.438307 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-home\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.438535 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.438332 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-dshm\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.438535 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.438404 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-tmp-dir\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.438624 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.438570 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049539c3-9a7f-4b23-b307-9e634705772a-tls-certs\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.438735 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.438716 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkxk2\" (UniqueName: \"kubernetes.io/projected/049539c3-9a7f-4b23-b307-9e634705772a-kube-api-access-vkxk2\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.490226 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.490168 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" event={"ID":"64e997e7-9d8b-469b-ac1f-743fa96f7e1f","Type":"ContainerDied","Data":"e272140713d6c8ed84fe42194961e6574987dd24a0aa6c4a73969ec0d9ffdf06"} Nov 03 21:45:21.490596 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.490572 5041 scope.go:122] "RemoveContainer" containerID="25ca0460bb219959e5ce2f6081c9d29bc4c2e079dd1c5d80947c25681612267c" Nov 03 21:45:21.490897 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.490877 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k" Nov 03 21:45:21.497025 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.496992 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" event={"ID":"c2b4096e-7ea0-4377-a8a0-67c3c27d0036","Type":"ContainerDied","Data":"3baed2721adbbe24463d6665a0845e7f63499f5f2785d5f065f71bb25e21cc33"} Nov 03 21:45:21.504917 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.504642 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" event={"ID":"947188bc-8b80-416c-b31a-c3ed370dd012","Type":"ContainerDied","Data":"e2173bcd8d4fcf5557312144b66b135fd016ed6d1e7a3cb4e5e1d02d77787a49"} Nov 03 21:45:21.505167 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.502718 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl" Nov 03 21:45:21.505602 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.497244 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs" Nov 03 21:45:21.541602 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.539915 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkxk2\" (UniqueName: \"kubernetes.io/projected/049539c3-9a7f-4b23-b307-9e634705772a-kube-api-access-vkxk2\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.541602 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.539999 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-model-cache\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.541602 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.540029 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-home\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.541602 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.540057 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-dshm\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.541602 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.540080 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-tmp-dir\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.541602 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.540166 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049539c3-9a7f-4b23-b307-9e634705772a-tls-certs\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.541602 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.541043 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-home\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.542082 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.541646 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-tmp-dir\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.542082 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.541976 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-model-cache\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.544253 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.543901 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049539c3-9a7f-4b23-b307-9e634705772a-tls-certs\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.546655 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.546633 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-dshm\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.555014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.554992 5041 scope.go:122] "RemoveContainer" containerID="7727916f85b006da355e378b8e3f17561402d0dbade80268d909f2e57f038d5a" Nov 03 21:45:21.565596 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.565570 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkxk2\" (UniqueName: \"kubernetes.io/projected/049539c3-9a7f-4b23-b307-9e634705772a-kube-api-access-vkxk2\") pod \"distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.593421 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.593384 5041 scope.go:122] "RemoveContainer" containerID="a601e09f6a15dc97af0d5ff65d132e5b5ea714097748dd1b66c8eb9a37723028" Nov 03 21:45:21.607138 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.606971 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:21.653291 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.653244 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k"] Nov 03 21:45:21.657885 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.657858 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-8ct6k"] Nov 03 21:45:21.690892 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.690799 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl"] Nov 03 21:45:21.697584 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.697555 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e246f9-29895705-68svl"] Nov 03 21:45:21.728749 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.728720 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs"] Nov 03 21:45:21.730258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.730235 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" Nov 03 21:45:21.734656 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.734629 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27fdf4-29895705-sqrqs"] Nov 03 21:45:21.848008 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.847926 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43eed4a-92c9-4074-936f-b48712db3dfc-kube-api-access-bcp8n\" (UniqueName: \"kubernetes.io/projected/f43eed4a-92c9-4074-936f-b48712db3dfc-kube-api-access-bcp8n\") pod \"f43eed4a-92c9-4074-936f-b48712db3dfc\" (UID: \"f43eed4a-92c9-4074-936f-b48712db3dfc\") " Nov 03 21:45:21.848512 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.848463 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43eed4a-92c9-4074-936f-b48712db3dfc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43eed4a-92c9-4074-936f-b48712db3dfc-maas-api-service-ca\") pod \"f43eed4a-92c9-4074-936f-b48712db3dfc\" (UID: \"f43eed4a-92c9-4074-936f-b48712db3dfc\") " Nov 03 21:45:21.848990 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.848929 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43eed4a-92c9-4074-936f-b48712db3dfc-maas-api-service-ca" pod "f43eed4a-92c9-4074-936f-b48712db3dfc" (UID: "f43eed4a-92c9-4074-936f-b48712db3dfc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:21.849703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.849625 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43eed4a-92c9-4074-936f-b48712db3dfc-maas-api-service-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:21.852704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.850949 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43eed4a-92c9-4074-936f-b48712db3dfc-kube-api-access-bcp8n" pod "f43eed4a-92c9-4074-936f-b48712db3dfc" (UID: "f43eed4a-92c9-4074-936f-b48712db3dfc"). InnerVolumeSpecName "kube-api-access-bcp8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:21.937553 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.937524 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs"] Nov 03 21:45:21.938942 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:45:21.938899 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod049539c3_9a7f_4b23_b307_9e634705772a.slice/crio-68f993a50b84d007639cf0ebedb922cc2560f3c67b39d1cb7846f5e253a4cf92 WatchSource:0}: Error finding container 68f993a50b84d007639cf0ebedb922cc2560f3c67b39d1cb7846f5e253a4cf92: Status 404 returned error can't find the container with id 68f993a50b84d007639cf0ebedb922cc2560f3c67b39d1cb7846f5e253a4cf92 Nov 03 21:45:21.950555 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:21.950530 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcp8n\" (UniqueName: \"kubernetes.io/projected/f43eed4a-92c9-4074-936f-b48712db3dfc-kube-api-access-bcp8n\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:22.547005 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:22.546973 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" Nov 03 21:45:22.547005 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:22.546988 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7" event={"ID":"f43eed4a-92c9-4074-936f-b48712db3dfc","Type":"ContainerDied","Data":"b3649f5f4fe78849683806f4a6b079ab3d137e0b8bd3fdb167b210ea9f732e7c"} Nov 03 21:45:22.547242 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:22.547091 5041 scope.go:122] "RemoveContainer" containerID="8ff5fa6ac19a0764f4aa573547945f50bcfd0125910e86d17a21d13fb2e5a9c2" Nov 03 21:45:22.551884 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:22.551851 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" event={"ID":"049539c3-9a7f-4b23-b307-9e634705772a","Type":"ContainerStarted","Data":"faa7e30f9a287f45a4856e7d2032e5f055df733a1d3df587d6f983f9e7387551"} Nov 03 21:45:22.551985 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:22.551900 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" event={"ID":"049539c3-9a7f-4b23-b307-9e634705772a","Type":"ContainerStarted","Data":"68f993a50b84d007639cf0ebedb922cc2560f3c67b39d1cb7846f5e253a4cf92"} Nov 03 21:45:22.605804 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:22.605775 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7"] Nov 03 21:45:22.607301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:22.607277 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:22.625312 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:22.625283 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e997e7-9d8b-469b-ac1f-743fa96f7e1f" path="/var/lib/kubelet/pods/64e997e7-9d8b-469b-ac1f-743fa96f7e1f/volumes" Nov 03 21:45:22.625771 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:22.625754 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="947188bc-8b80-416c-b31a-c3ed370dd012" path="/var/lib/kubelet/pods/947188bc-8b80-416c-b31a-c3ed370dd012/volumes" Nov 03 21:45:22.626139 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:22.626124 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b4096e-7ea0-4377-a8a0-67c3c27d0036" path="/var/lib/kubelet/pods/c2b4096e-7ea0-4377-a8a0-67c3c27d0036/volumes" Nov 03 21:45:22.626788 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:22.626770 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4eced5-29895705-mb6d7"] Nov 03 21:45:22.651578 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:22.651532 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podStartSLOduration=1.6515197019999999 podStartE2EDuration="1.651519702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:22.648444493 +0000 UTC m=+938.155672796" watchObservedRunningTime="2026-11-03 21:45:22.651519702 +0000 UTC m=+938.158747982" Nov 03 21:45:24.627310 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:24.627271 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43eed4a-92c9-4074-936f-b48712db3dfc" path="/var/lib/kubelet/pods/f43eed4a-92c9-4074-936f-b48712db3dfc/volumes" Nov 03 21:45:25.232079 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.232031 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg"] Nov 03 21:45:25.233006 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.232979 5041 cpu_manager.go:414] "containerMap: removing container" podUID="832c2242-e330-4541-961b-4acde0fbd38f" containerName="cleanup" Nov 03 21:45:25.233006 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.233005 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832c2242-e330-4541-961b-4acde0fbd38f" containerName="cleanup" Nov 03 21:45:25.233180 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.233025 5041 cpu_manager.go:414] "containerMap: removing container" podUID="f43eed4a-92c9-4074-936f-b48712db3dfc" containerName="cleanup" Nov 03 21:45:25.233180 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.233034 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43eed4a-92c9-4074-936f-b48712db3dfc" containerName="cleanup" Nov 03 21:45:25.233180 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.233060 5041 cpu_manager.go:414] "containerMap: removing container" podUID="f43eed4a-92c9-4074-936f-b48712db3dfc" containerName="cleanup" Nov 03 21:45:25.233180 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.233069 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43eed4a-92c9-4074-936f-b48712db3dfc" containerName="cleanup" Nov 03 21:45:25.233180 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.233084 5041 cpu_manager.go:414] "containerMap: removing container" podUID="64e997e7-9d8b-469b-ac1f-743fa96f7e1f" containerName="cleanup" Nov 03 21:45:25.233180 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.233089 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e997e7-9d8b-469b-ac1f-743fa96f7e1f" containerName="cleanup" Nov 03 21:45:25.233180 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.233101 5041 cpu_manager.go:414] "containerMap: removing container" podUID="f43eed4a-92c9-4074-936f-b48712db3dfc" containerName="cleanup" Nov 03 21:45:25.233180 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.233109 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43eed4a-92c9-4074-936f-b48712db3dfc" containerName="cleanup" Nov 03 21:45:25.233503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.233278 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e997e7-9d8b-469b-ac1f-743fa96f7e1f" containerName="cleanup" Nov 03 21:45:25.233503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.233300 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="832c2242-e330-4541-961b-4acde0fbd38f" containerName="cleanup" Nov 03 21:45:25.233503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.233311 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43eed4a-92c9-4074-936f-b48712db3dfc" containerName="cleanup" Nov 03 21:45:25.233503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.233321 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43eed4a-92c9-4074-936f-b48712db3dfc" containerName="cleanup" Nov 03 21:45:25.233503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.233332 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43eed4a-92c9-4074-936f-b48712db3dfc" containerName="cleanup" Nov 03 21:45:25.239080 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.239054 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.262878 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.261177 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg"] Nov 03 21:45:25.402712 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.401633 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f738b002-3e77-4a00-9982-2cb525d164e6-istio-podinfo\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.402868 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.402750 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v2z5\" (UniqueName: \"kubernetes.io/projected/f738b002-3e77-4a00-9982-2cb525d164e6-kube-api-access-9v2z5\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.402868 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.402790 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-workload-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.403006 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.402982 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f738b002-3e77-4a00-9982-2cb525d164e6-istio-token\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.403117 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.403100 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-credential-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.403157 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.403133 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-istio-data\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.403214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.403154 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f738b002-3e77-4a00-9982-2cb525d164e6-istiod-ca-cert\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.403353 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.403323 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-istio-envoy\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.403471 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.403437 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-workload-certs\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.504579 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.504479 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-credential-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.504579 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.504519 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-istio-data\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.504579 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.504548 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f738b002-3e77-4a00-9982-2cb525d164e6-istiod-ca-cert\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.504919 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.504624 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-istio-envoy\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.504919 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.504724 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-workload-certs\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.504919 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.504763 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f738b002-3e77-4a00-9982-2cb525d164e6-istio-podinfo\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.504919 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.504800 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v2z5\" (UniqueName: \"kubernetes.io/projected/f738b002-3e77-4a00-9982-2cb525d164e6-kube-api-access-9v2z5\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.504919 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.504845 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-workload-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.505169 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.504973 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f738b002-3e77-4a00-9982-2cb525d164e6-istio-token\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.505169 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.505006 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-istio-data\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.505169 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.505116 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-workload-certs\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.505352 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.505321 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-workload-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.505412 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.505355 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f738b002-3e77-4a00-9982-2cb525d164e6-istiod-ca-cert\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.505471 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.505447 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-credential-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.507285 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.507261 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-istio-envoy\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.507634 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.507615 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f738b002-3e77-4a00-9982-2cb525d164e6-istio-podinfo\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.528983 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.528947 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f738b002-3e77-4a00-9982-2cb525d164e6-istio-token\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.529478 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.529456 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v2z5\" (UniqueName: \"kubernetes.io/projected/f738b002-3e77-4a00-9982-2cb525d164e6-kube-api-access-9v2z5\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.565807 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.565751 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:25.896107 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.895043 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg"] Nov 03 21:45:25.898578 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:45:25.898549 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf738b002_3e77_4a00_9982_2cb525d164e6.slice/crio-8ccf964d0453c65561a05cfd52897ed82b765147372293bc96bbe62a6cd4ff0a WatchSource:0}: Error finding container 8ccf964d0453c65561a05cfd52897ed82b765147372293bc96bbe62a6cd4ff0a: Status 404 returned error can't find the container with id 8ccf964d0453c65561a05cfd52897ed82b765147372293bc96bbe62a6cd4ff0a Nov 03 21:45:25.901554 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.901521 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:25.901629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.901588 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:25.901629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:25.901617 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:26.593915 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:26.593880 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" event={"ID":"f738b002-3e77-4a00-9982-2cb525d164e6","Type":"ContainerStarted","Data":"85b11d0a2404d3902b7fda4df0d136d596953392540e87ec21dfd80040c158e5"} Nov 03 21:45:26.594099 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:26.593943 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" event={"ID":"f738b002-3e77-4a00-9982-2cb525d164e6","Type":"ContainerStarted","Data":"8ccf964d0453c65561a05cfd52897ed82b765147372293bc96bbe62a6cd4ff0a"} Nov 03 21:45:26.672620 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:26.672565 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" podStartSLOduration=1.672550372 podStartE2EDuration="1.672550372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:26.655374745 +0000 UTC m=+942.162603027" watchObservedRunningTime="2026-11-03 21:45:26.672550372 +0000 UTC m=+942.179778646" Nov 03 21:45:27.566638 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:27.566606 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:27.568124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:27.568096 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:45:27.568282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:27.568151 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" podUID="f738b002-3e77-4a00-9982-2cb525d164e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 21:45:28.566465 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:28.566429 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:45:28.566634 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:28.566482 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" podUID="f738b002-3e77-4a00-9982-2cb525d164e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 21:45:29.566647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:29.566611 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:45:29.567048 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:29.566699 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" podUID="f738b002-3e77-4a00-9982-2cb525d164e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 21:45:30.567081 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:30.567044 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:45:30.567509 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:30.567100 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" podUID="f738b002-3e77-4a00-9982-2cb525d164e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 21:45:31.566750 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:31.566708 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:45:31.567017 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:31.566765 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" podUID="f738b002-3e77-4a00-9982-2cb525d164e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 21:45:31.619696 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:31.618049 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:31.728062 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:31.728021 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:31.886142 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:31.886044 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:32.568139 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:32.568103 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:45:32.568428 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:32.568398 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" podUID="f738b002-3e77-4a00-9982-2cb525d164e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 21:45:32.985812 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:32.985762 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:45:33.321747 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:33.313304 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:45:33.498712 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:33.495794 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:45:33.700880 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:33.700767 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:45:33.782892 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:33.782845 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:45:33.993209 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:33.993162 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:45:33.994516 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:33.994489 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:45:34.049894 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:34.049850 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:45:34.358747 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:34.348814 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:45:34.447686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:34.363390 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:34.519789 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:34.519745 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:45:34.521258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:34.521232 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:45:34.611189 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:34.611091 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:45:34.618787 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:34.618739 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" podUID="f738b002-3e77-4a00-9982-2cb525d164e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:45:34.784686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:34.784471 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:45:34.926221 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:34.926126 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:34.926521 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:34.926494 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:45:34.998785 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:34.998752 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:45:35.000473 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:35.000436 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:45:35.100973 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:35.100943 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:35.102536 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:35.102514 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:35.104457 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:35.104443 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:45:35.358950 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:35.358903 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:45:35.446697 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:35.446028 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:45:35.454163 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:35.454134 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:45:40.777534 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:45:40.777496 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:45:42.144025 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:42.143990 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt"] Nov 03 21:45:42.149977 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:42.149956 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt" Nov 03 21:45:42.166635 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:42.166607 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt"] Nov 03 21:45:42.173942 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:42.173914 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:45:42.174090 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:42.173981 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:45:42.296063 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:42.296028 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35091bce-d209-4b19-a4f3-ce7707badfad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b20ba06-ll5mt\" (UID: \"35091bce-d209-4b19-a4f3-ce7707badfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt" Nov 03 21:45:42.296254 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:42.296082 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35091bce-d209-4b19-a4f3-ce7707badfad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b20ba06-ll5mt\" (UID: \"35091bce-d209-4b19-a4f3-ce7707badfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt" Nov 03 21:45:42.296333 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:42.296308 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpn7p\" (UniqueName: \"kubernetes.io/projected/35091bce-d209-4b19-a4f3-ce7707badfad-kube-api-access-jpn7p\") pod \"maas-api-revoke-sub-6b20ba06-ll5mt\" (UID: \"35091bce-d209-4b19-a4f3-ce7707badfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt" Nov 03 21:45:42.398284 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:42.398205 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35091bce-d209-4b19-a4f3-ce7707badfad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b20ba06-ll5mt\" (UID: \"35091bce-d209-4b19-a4f3-ce7707badfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt" Nov 03 21:45:42.398284 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:42.398267 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35091bce-d209-4b19-a4f3-ce7707badfad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b20ba06-ll5mt\" (UID: \"35091bce-d209-4b19-a4f3-ce7707badfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt" Nov 03 21:45:42.398527 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:42.398436 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpn7p\" (UniqueName: \"kubernetes.io/projected/35091bce-d209-4b19-a4f3-ce7707badfad-kube-api-access-jpn7p\") pod \"maas-api-revoke-sub-6b20ba06-ll5mt\" (UID: \"35091bce-d209-4b19-a4f3-ce7707badfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt" Nov 03 21:45:42.398999 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:42.398975 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35091bce-d209-4b19-a4f3-ce7707badfad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b20ba06-ll5mt\" (UID: \"35091bce-d209-4b19-a4f3-ce7707badfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt" Nov 03 21:45:42.400808 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:42.400781 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35091bce-d209-4b19-a4f3-ce7707badfad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b20ba06-ll5mt\" (UID: \"35091bce-d209-4b19-a4f3-ce7707badfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt" Nov 03 21:45:42.418534 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:42.418505 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpn7p\" (UniqueName: \"kubernetes.io/projected/35091bce-d209-4b19-a4f3-ce7707badfad-kube-api-access-jpn7p\") pod \"maas-api-revoke-sub-6b20ba06-ll5mt\" (UID: \"35091bce-d209-4b19-a4f3-ce7707badfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt" Nov 03 21:45:42.468927 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:42.468901 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt" Nov 03 21:45:42.772024 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:45:42.771995 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35091bce_d209_4b19_a4f3_ce7707badfad.slice/crio-4d2d32a28f4919c5ecf0e8df5dc864fcdeccef088beec5f5a560dcc3e3d7c1de WatchSource:0}: Error finding container 4d2d32a28f4919c5ecf0e8df5dc864fcdeccef088beec5f5a560dcc3e3d7c1de: Status 404 returned error can't find the container with id 4d2d32a28f4919c5ecf0e8df5dc864fcdeccef088beec5f5a560dcc3e3d7c1de Nov 03 21:45:42.772985 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:42.772877 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt"] Nov 03 21:45:43.006268 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:43.006223 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt" event={"ID":"35091bce-d209-4b19-a4f3-ce7707badfad","Type":"ContainerStarted","Data":"4d2d32a28f4919c5ecf0e8df5dc864fcdeccef088beec5f5a560dcc3e3d7c1de"} Nov 03 21:45:44.027725 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:44.027591 5041 generic.go:356] "Generic (PLEG): container finished" podID="35091bce-d209-4b19-a4f3-ce7707badfad" containerID="7ed16b6cd4703f556c23abb841135e54604090021fc1968728b64a9dffdeedb6" exitCode=0 Nov 03 21:45:44.027725 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:44.027652 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt" event={"ID":"35091bce-d209-4b19-a4f3-ce7707badfad","Type":"ContainerDied","Data":"7ed16b6cd4703f556c23abb841135e54604090021fc1968728b64a9dffdeedb6"} Nov 03 21:45:45.225285 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:45.225256 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt" Nov 03 21:45:45.243583 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:45.243558 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35091bce-d209-4b19-a4f3-ce7707badfad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35091bce-d209-4b19-a4f3-ce7707badfad-maas-api-xks-ca\") pod \"35091bce-d209-4b19-a4f3-ce7707badfad\" (UID: \"35091bce-d209-4b19-a4f3-ce7707badfad\") " Nov 03 21:45:45.243718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:45.243594 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35091bce-d209-4b19-a4f3-ce7707badfad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35091bce-d209-4b19-a4f3-ce7707badfad-maas-api-openshift-ca\") pod \"35091bce-d209-4b19-a4f3-ce7707badfad\" (UID: \"35091bce-d209-4b19-a4f3-ce7707badfad\") " Nov 03 21:45:45.243783 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:45.243766 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35091bce-d209-4b19-a4f3-ce7707badfad-kube-api-access-jpn7p\" (UniqueName: \"kubernetes.io/projected/35091bce-d209-4b19-a4f3-ce7707badfad-kube-api-access-jpn7p\") pod \"35091bce-d209-4b19-a4f3-ce7707badfad\" (UID: \"35091bce-d209-4b19-a4f3-ce7707badfad\") " Nov 03 21:45:45.244018 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:45.243976 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35091bce-d209-4b19-a4f3-ce7707badfad-maas-api-openshift-ca" pod "35091bce-d209-4b19-a4f3-ce7707badfad" (UID: "35091bce-d209-4b19-a4f3-ce7707badfad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:45:45.244341 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:45.244323 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35091bce-d209-4b19-a4f3-ce7707badfad-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:45.245870 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:45.245846 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35091bce-d209-4b19-a4f3-ce7707badfad-kube-api-access-jpn7p" pod "35091bce-d209-4b19-a4f3-ce7707badfad" (UID: "35091bce-d209-4b19-a4f3-ce7707badfad"). InnerVolumeSpecName "kube-api-access-jpn7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:45:45.245958 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:45.245849 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35091bce-d209-4b19-a4f3-ce7707badfad-maas-api-xks-ca" pod "35091bce-d209-4b19-a4f3-ce7707badfad" (UID: "35091bce-d209-4b19-a4f3-ce7707badfad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:45:45.345406 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:45.345315 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35091bce-d209-4b19-a4f3-ce7707badfad-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:45.345406 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:45.345343 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpn7p\" (UniqueName: \"kubernetes.io/projected/35091bce-d209-4b19-a4f3-ce7707badfad-kube-api-access-jpn7p\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:45:46.045079 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:46.045040 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt" event={"ID":"35091bce-d209-4b19-a4f3-ce7707badfad","Type":"ContainerDied","Data":"4d2d32a28f4919c5ecf0e8df5dc864fcdeccef088beec5f5a560dcc3e3d7c1de"} Nov 03 21:45:46.045273 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:46.045106 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d2d32a28f4919c5ecf0e8df5dc864fcdeccef088beec5f5a560dcc3e3d7c1de" Nov 03 21:45:46.045273 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:46.045054 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt" Nov 03 21:45:55.056191 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.056151 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg"] Nov 03 21:45:55.057498 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.057396 5041 cpu_manager.go:414] "containerMap: removing container" podUID="35091bce-d209-4b19-a4f3-ce7707badfad" containerName="revoke-keys" Nov 03 21:45:55.057498 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.057421 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35091bce-d209-4b19-a4f3-ce7707badfad" containerName="revoke-keys" Nov 03 21:45:55.057708 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.057622 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="35091bce-d209-4b19-a4f3-ce7707badfad" containerName="revoke-keys" Nov 03 21:45:55.063139 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.063118 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.092705 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.087971 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg"] Nov 03 21:45:55.092705 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.090493 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb"] Nov 03 21:45:55.095630 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.095599 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.130568 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.130539 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb"] Nov 03 21:45:55.203704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.203330 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-workload-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.203704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.203408 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-istio-envoy\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.203704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.203438 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-token\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.203704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.203492 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-data\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.203704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.203594 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-workload-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.204054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.203744 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-istio-data\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.204054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.203817 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmxwf\" (UniqueName: \"kubernetes.io/projected/bc499793-421d-4550-af6d-691af2b07aa9-kube-api-access-pmxwf\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.204054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.203889 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc499793-421d-4550-af6d-691af2b07aa9-istio-podinfo\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.204054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.203980 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc499793-421d-4550-af6d-691af2b07aa9-istiod-ca-cert\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.204054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.204007 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-workload-certs\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.204054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.204047 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-workload-certs\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.204254 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.204089 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-credential-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.204254 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.204135 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-envoy\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.204254 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.204160 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-podinfo\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.204254 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.204190 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc499793-421d-4550-af6d-691af2b07aa9-istio-token\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.204254 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.204247 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istiod-ca-cert\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.204404 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.204297 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-credential-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.204404 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.204324 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsljr\" (UniqueName: \"kubernetes.io/projected/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-kube-api-access-hsljr\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.214170 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.214132 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm"] Nov 03 21:45:55.219817 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.219788 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.245221 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.244417 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm"] Nov 03 21:45:55.305207 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.305174 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-envoy\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.305207 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.305209 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-podinfo\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.305472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.305236 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc499793-421d-4550-af6d-691af2b07aa9-istio-token\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.305472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.305270 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lf2t\" (UniqueName: \"kubernetes.io/projected/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-kube-api-access-2lf2t\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.305472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.305300 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-workload-certs\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.305472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.305338 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istiod-ca-cert\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.305746 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.305517 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-credential-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.305746 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.305558 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsljr\" (UniqueName: \"kubernetes.io/projected/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-kube-api-access-hsljr\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.305746 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.305701 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-workload-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.305993 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.305973 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-credential-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.306079 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306036 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-workload-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.306263 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306199 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istiod-ca-cert\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.306263 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306218 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-istio-envoy\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.306263 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306254 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-token\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.306492 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306285 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-workload-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.306492 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306327 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-data\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.306492 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306352 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-data\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.306492 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306378 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istiod-ca-cert\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.306492 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306410 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-podinfo\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.306492 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306474 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-workload-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.306820 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306509 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-istio-data\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.306820 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306656 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-envoy\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.306820 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306784 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmxwf\" (UniqueName: \"kubernetes.io/projected/bc499793-421d-4550-af6d-691af2b07aa9-kube-api-access-pmxwf\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.306820 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306798 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-data\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.306946 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306865 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc499793-421d-4550-af6d-691af2b07aa9-istio-podinfo\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.306946 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306924 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-token\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.306946 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306934 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-istio-data\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.307033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.306977 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-credential-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.307033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.307024 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc499793-421d-4550-af6d-691af2b07aa9-istiod-ca-cert\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.307096 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.307050 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-workload-certs\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.307130 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.307098 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-workload-certs\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.307219 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.307194 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-credential-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.307361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.307339 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-workload-certs\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.307452 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.307421 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-workload-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.307753 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.307709 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-credential-socket\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.308038 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.308014 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc499793-421d-4550-af6d-691af2b07aa9-istiod-ca-cert\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.308283 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.308263 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-envoy\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.308459 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.308413 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-workload-certs\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.308581 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.308497 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-podinfo\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.309707 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.309407 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc499793-421d-4550-af6d-691af2b07aa9-istio-podinfo\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.309707 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.309604 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-istio-envoy\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.326804 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.326777 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsljr\" (UniqueName: \"kubernetes.io/projected/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-kube-api-access-hsljr\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.326941 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.326876 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-token\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.327471 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.327444 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmxwf\" (UniqueName: \"kubernetes.io/projected/bc499793-421d-4550-af6d-691af2b07aa9-kube-api-access-pmxwf\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.328025 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.328002 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc499793-421d-4550-af6d-691af2b07aa9-istio-token\") pod \"e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.377640 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.377614 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:55.408950 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.408912 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lf2t\" (UniqueName: \"kubernetes.io/projected/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-kube-api-access-2lf2t\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.409094 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.408964 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-workload-certs\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.409094 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.409088 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-workload-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.409209 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.409124 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-data\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.409209 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.409150 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istiod-ca-cert\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.409209 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.409177 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-podinfo\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.409412 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.409242 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-envoy\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.409412 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.409340 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-token\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.409412 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.409381 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-credential-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.409567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.409448 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-workload-certs\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.409626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.409589 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-data\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.410553 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.409864 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-workload-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.410553 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.410255 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-credential-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.410553 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.410508 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istiod-ca-cert\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.413243 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.413216 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-podinfo\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.413439 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.413417 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-envoy\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.427830 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.427801 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:55.429524 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.429498 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lf2t\" (UniqueName: \"kubernetes.io/projected/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-kube-api-access-2lf2t\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.429959 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.429942 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-token\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.535242 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.535029 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:55.719534 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:45:55.719497 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cb2586_4e9a_47cf_abd8_604c0f1264d5.slice/crio-55f6fe46d3ec76d314820469f16cbad3f895d85550d941686a67b0333479861f WatchSource:0}: Error finding container 55f6fe46d3ec76d314820469f16cbad3f895d85550d941686a67b0333479861f: Status 404 returned error can't find the container with id 55f6fe46d3ec76d314820469f16cbad3f895d85550d941686a67b0333479861f Nov 03 21:45:55.722248 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.722209 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:55.722377 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.722292 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:55.722377 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.722328 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:55.723822 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.723412 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg"] Nov 03 21:45:55.757687 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:45:55.757279 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc499793_421d_4550_af6d_691af2b07aa9.slice/crio-5c169014dbbb7393b5b4dc8e6c8e3fb341a928501e7cdaff29850d629d6c7449 WatchSource:0}: Error finding container 5c169014dbbb7393b5b4dc8e6c8e3fb341a928501e7cdaff29850d629d6c7449: Status 404 returned error can't find the container with id 5c169014dbbb7393b5b4dc8e6c8e3fb341a928501e7cdaff29850d629d6c7449 Nov 03 21:45:55.757687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.757564 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb"] Nov 03 21:45:55.760185 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.760154 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:55.760267 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.760215 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:55.760267 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.760246 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:55.872147 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.871942 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm"] Nov 03 21:45:55.873004 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:45:55.872974 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29a22a0_dc41_4a86_ac9f_d6392e6c5472.slice/crio-961a0747646ad75f831c49801a1c46453f1b90c2075708b397fc12c478f7d506 WatchSource:0}: Error finding container 961a0747646ad75f831c49801a1c46453f1b90c2075708b397fc12c478f7d506: Status 404 returned error can't find the container with id 961a0747646ad75f831c49801a1c46453f1b90c2075708b397fc12c478f7d506 Nov 03 21:45:55.876903 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.876852 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:55.876989 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.876970 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:55.877047 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:55.877017 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:45:56.200597 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.200502 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" event={"ID":"e9cb2586-4e9a-47cf-abd8-604c0f1264d5","Type":"ContainerStarted","Data":"2ec44d11eb69ed51556dc1bf695e17e85f744689e052fd8945b6dc82833f5f8e"} Nov 03 21:45:56.200597 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.200560 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" event={"ID":"e9cb2586-4e9a-47cf-abd8-604c0f1264d5","Type":"ContainerStarted","Data":"55f6fe46d3ec76d314820469f16cbad3f895d85550d941686a67b0333479861f"} Nov 03 21:45:56.202920 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.202891 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" event={"ID":"e29a22a0-dc41-4a86-ac9f-d6392e6c5472","Type":"ContainerStarted","Data":"18aebf2043dd1aed5e66119ec2b656a4f4d475acbc28e4fabbc675b3aafa73d5"} Nov 03 21:45:56.203114 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.202943 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" event={"ID":"e29a22a0-dc41-4a86-ac9f-d6392e6c5472","Type":"ContainerStarted","Data":"961a0747646ad75f831c49801a1c46453f1b90c2075708b397fc12c478f7d506"} Nov 03 21:45:56.205308 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.205284 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" event={"ID":"bc499793-421d-4550-af6d-691af2b07aa9","Type":"ContainerStarted","Data":"87f72ddcebb07a3581535f3bab87e30edbc284398a8b3bf852027963aff7a98d"} Nov 03 21:45:56.205424 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.205326 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" event={"ID":"bc499793-421d-4550-af6d-691af2b07aa9","Type":"ContainerStarted","Data":"5c169014dbbb7393b5b4dc8e6c8e3fb341a928501e7cdaff29850d629d6c7449"} Nov 03 21:45:56.259999 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.259929 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" podStartSLOduration=1.25990922 podStartE2EDuration="1.25990922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:56.250717768 +0000 UTC m=+971.757946045" watchObservedRunningTime="2026-11-03 21:45:56.25990922 +0000 UTC m=+971.767137502" Nov 03 21:45:56.270203 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.269931 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4"] Nov 03 21:45:56.274924 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.274898 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.293881 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.293847 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4"] Nov 03 21:45:56.294351 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.294331 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e246f9\"/\"kube-root-ca.crt\"" Nov 03 21:45:56.294628 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.294614 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e246f9\"/\"default-dockercfg-pvxz5\"" Nov 03 21:45:56.294847 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.294833 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e246f9\"/\"distinct-2-w0-e246f9-kserve-self-signed-certs\"" Nov 03 21:45:56.295013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.295001 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e246f9\"/\"openshift-service-ca.crt\"" Nov 03 21:45:56.307428 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.307332 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" podStartSLOduration=1.3073142309999999 podStartE2EDuration="1.307314231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:56.301926903 +0000 UTC m=+971.809155187" watchObservedRunningTime="2026-11-03 21:45:56.307314231 +0000 UTC m=+971.814542512" Nov 03 21:45:56.348396 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.348334 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" podStartSLOduration=1.348321037 podStartE2EDuration="1.348321037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:56.34640273 +0000 UTC m=+971.853631038" watchObservedRunningTime="2026-11-03 21:45:56.348321037 +0000 UTC m=+971.855549318" Nov 03 21:45:56.379273 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.379236 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:45:56.382719 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.380852 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:45:56.382719 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.380914 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" podUID="e9cb2586-4e9a-47cf-abd8-604c0f1264d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 21:45:56.428286 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.428206 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:45:56.430365 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.430331 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:45:56.430508 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.430394 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" podUID="bc499793-421d-4550-af6d-691af2b07aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 21:45:56.436014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.435358 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-model-cache\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.436014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.435582 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9448e92-6c58-4dfe-93e9-7dc304b97066-tls-certs\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.436014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.435622 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-dshm\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.436014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.435718 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-tmp-dir\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.436014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.435768 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-home\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.436014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.435794 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k6fz\" (UniqueName: \"kubernetes.io/projected/c9448e92-6c58-4dfe-93e9-7dc304b97066-kube-api-access-9k6fz\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.550691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.540853 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-model-cache\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.550691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.541002 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9448e92-6c58-4dfe-93e9-7dc304b97066-tls-certs\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.550691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.541041 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-dshm\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.550691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.541124 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-tmp-dir\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.550691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.541173 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-home\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.550691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.541198 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k6fz\" (UniqueName: \"kubernetes.io/projected/c9448e92-6c58-4dfe-93e9-7dc304b97066-kube-api-access-9k6fz\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.550691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.541998 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-model-cache\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.565684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.551570 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-tmp-dir\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.565979 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.556070 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-home\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.566106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.558304 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9448e92-6c58-4dfe-93e9-7dc304b97066-tls-certs\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.566203 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.561528 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:45:56.566337 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.561601 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:45:56.566468 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.566442 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" podUID="e29a22a0-dc41-4a86-ac9f-d6392e6c5472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 21:45:56.575577 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.575511 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k6fz\" (UniqueName: \"kubernetes.io/projected/c9448e92-6c58-4dfe-93e9-7dc304b97066-kube-api-access-9k6fz\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.587062 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.587032 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-dshm\") pod \"distinct-2-w0-e246f9-kserve-64869f999d-qtdj4\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:56.678761 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:56.674015 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:57.506782 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:57.482293 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:45:57.506782 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:57.482357 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" podUID="e9cb2586-4e9a-47cf-abd8-604c0f1264d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 21:45:57.506782 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:57.482981 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:45:57.506782 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:57.483018 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" podUID="bc499793-421d-4550-af6d-691af2b07aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 21:45:57.547555 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:57.543589 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:45:57.547555 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:57.543684 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" podUID="e29a22a0-dc41-4a86-ac9f-d6392e6c5472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 21:45:57.918278 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:45:57.916567 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9448e92_6c58_4dfe_93e9_7dc304b97066.slice/crio-1664fee5984ad0d413f91170b69f703579e1bf50a89304f162c788324f8dccb5 WatchSource:0}: Error finding container 1664fee5984ad0d413f91170b69f703579e1bf50a89304f162c788324f8dccb5: Status 404 returned error can't find the container with id 1664fee5984ad0d413f91170b69f703579e1bf50a89304f162c788324f8dccb5 Nov 03 21:45:57.919987 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:57.919794 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4"] Nov 03 21:45:58.313139 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:58.309904 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" event={"ID":"c9448e92-6c58-4dfe-93e9-7dc304b97066","Type":"ContainerStarted","Data":"1664fee5984ad0d413f91170b69f703579e1bf50a89304f162c788324f8dccb5"} Nov 03 21:45:58.386686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:58.381060 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:45:58.386686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:58.381125 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" podUID="e9cb2586-4e9a-47cf-abd8-604c0f1264d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 21:45:58.435689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:58.434579 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:45:58.549728 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:58.434648 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" podUID="bc499793-421d-4550-af6d-691af2b07aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 21:45:58.591769 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:58.591049 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:45:58.591769 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:58.591110 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" podUID="e29a22a0-dc41-4a86-ac9f-d6392e6c5472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 21:45:59.357686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:59.337728 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" event={"ID":"c9448e92-6c58-4dfe-93e9-7dc304b97066","Type":"ContainerStarted","Data":"2dc9f0723d698dfd82aa2d67882b0c0000ca3187d3d9b2208bb77a9cc7c80732"} Nov 03 21:45:59.383708 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:59.378005 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:45:59.383708 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:59.378066 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" podUID="e9cb2586-4e9a-47cf-abd8-604c0f1264d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 21:45:59.450689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:59.450250 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:45:59.450689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:59.450310 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" podUID="bc499793-421d-4550-af6d-691af2b07aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 21:45:59.474685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:59.467998 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podStartSLOduration=3.467971397 podStartE2EDuration="3.467971397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:59.391951229 +0000 UTC m=+974.899179510" watchObservedRunningTime="2026-11-03 21:45:59.467971397 +0000 UTC m=+974.975199678" Nov 03 21:45:59.561690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:59.543215 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:45:59.561690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:59.543286 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" podUID="e29a22a0-dc41-4a86-ac9f-d6392e6c5472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 21:45:59.687866 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:59.687767 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:45:59.936812 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:45:59.936743 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:00.393255 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:00.390993 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:46:00.393255 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:00.391057 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" podUID="e9cb2586-4e9a-47cf-abd8-604c0f1264d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 21:46:00.448824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:00.436104 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:46:00.448824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:00.436172 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" podUID="bc499793-421d-4550-af6d-691af2b07aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 21:46:00.559448 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:00.559407 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:46:00.559656 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:00.559468 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" podUID="e29a22a0-dc41-4a86-ac9f-d6392e6c5472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 21:46:01.396706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:01.383083 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:46:01.396706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:01.383149 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" podUID="e9cb2586-4e9a-47cf-abd8-604c0f1264d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 21:46:01.432686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:01.431876 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:46:01.432686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:01.431953 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" podUID="bc499793-421d-4550-af6d-691af2b07aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 21:46:01.545687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:01.545149 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:46:01.545687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:01.545212 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" podUID="e29a22a0-dc41-4a86-ac9f-d6392e6c5472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 21:46:02.381689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:02.378169 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:46:02.381689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:02.378232 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" podUID="e9cb2586-4e9a-47cf-abd8-604c0f1264d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 21:46:02.428687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:02.428619 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:46:02.429339 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:02.429294 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" podUID="bc499793-421d-4550-af6d-691af2b07aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 21:46:02.537081 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:02.537042 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:46:02.537498 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:02.537468 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" podUID="e29a22a0-dc41-4a86-ac9f-d6392e6c5472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 21:46:03.378887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:03.378843 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:46:03.379221 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:03.378919 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" podUID="e9cb2586-4e9a-47cf-abd8-604c0f1264d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 21:46:03.428981 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:03.428939 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 21:46:03.429388 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:03.429012 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" podUID="bc499793-421d-4550-af6d-691af2b07aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 21:46:03.536068 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:03.536032 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 21:46:03.536359 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:03.536328 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" podUID="e29a22a0-dc41-4a86-ac9f-d6392e6c5472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 21:46:04.383144 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:04.383111 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:46:04.383556 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:04.383533 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:46:04.384276 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:04.384259 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:46:04.434115 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:04.434082 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:46:04.434616 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:04.434441 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:46:04.435387 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:04.435363 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:46:04.542721 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:04.542690 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:46:04.544223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:04.543438 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:46:04.545109 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:04.545086 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:46:06.684727 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:06.684692 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:46:06.700212 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:06.700183 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:46:06.791506 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:06.791474 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:46:10.779720 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:46:10.779649 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:46:15.915493 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:15.910869 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59"] Nov 03 21:46:15.921997 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:15.921963 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" Nov 03 21:46:15.966685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:15.943119 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59"] Nov 03 21:46:15.967037 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:15.944604 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:46:15.968169 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:15.960993 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:46:16.083277 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:16.068377 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd0a1d5-d575-4352-b513-130a585dbeec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1072f064-4dx59\" (UID: \"ffd0a1d5-d575-4352-b513-130a585dbeec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" Nov 03 21:46:16.099401 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:16.099352 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt5vs\" (UniqueName: \"kubernetes.io/projected/ffd0a1d5-d575-4352-b513-130a585dbeec-kube-api-access-mt5vs\") pod \"maas-api-revoke-sub-1072f064-4dx59\" (UID: \"ffd0a1d5-d575-4352-b513-130a585dbeec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" Nov 03 21:46:16.099861 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:16.099836 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd0a1d5-d575-4352-b513-130a585dbeec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1072f064-4dx59\" (UID: \"ffd0a1d5-d575-4352-b513-130a585dbeec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" Nov 03 21:46:16.221686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:16.221039 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt5vs\" (UniqueName: \"kubernetes.io/projected/ffd0a1d5-d575-4352-b513-130a585dbeec-kube-api-access-mt5vs\") pod \"maas-api-revoke-sub-1072f064-4dx59\" (UID: \"ffd0a1d5-d575-4352-b513-130a585dbeec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" Nov 03 21:46:16.221686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:16.221170 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd0a1d5-d575-4352-b513-130a585dbeec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1072f064-4dx59\" (UID: \"ffd0a1d5-d575-4352-b513-130a585dbeec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" Nov 03 21:46:16.221686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:16.221340 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd0a1d5-d575-4352-b513-130a585dbeec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1072f064-4dx59\" (UID: \"ffd0a1d5-d575-4352-b513-130a585dbeec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" Nov 03 21:46:16.285611 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:16.273532 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd0a1d5-d575-4352-b513-130a585dbeec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1072f064-4dx59\" (UID: \"ffd0a1d5-d575-4352-b513-130a585dbeec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" Nov 03 21:46:16.420686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:16.420138 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd0a1d5-d575-4352-b513-130a585dbeec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1072f064-4dx59\" (UID: \"ffd0a1d5-d575-4352-b513-130a585dbeec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" Nov 03 21:46:16.467963 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:16.467881 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt5vs\" (UniqueName: \"kubernetes.io/projected/ffd0a1d5-d575-4352-b513-130a585dbeec-kube-api-access-mt5vs\") pod \"maas-api-revoke-sub-1072f064-4dx59\" (UID: \"ffd0a1d5-d575-4352-b513-130a585dbeec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" Nov 03 21:46:16.613081 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:16.612976 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" Nov 03 21:46:17.113927 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:17.113818 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:17.398813 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:17.370925 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:17.715824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:17.715773 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:17.719684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:17.716138 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/health\": context deadline exceeded" Nov 03 21:46:17.719684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:17.716206 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 21:46:17.858719 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:17.833571 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 03 21:46:17.858719 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:17.833825 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:18.028034 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:18.027924 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:46:18.100074 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:18.098967 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:46:18.145690 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:46:18.140501 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd0a1d5_d575_4352_b513_130a585dbeec.slice/crio-379ab7aff5480d1cb94642e3a0ed055d95ad1c57be1b08bd095461e5d1a90157 WatchSource:0}: Error finding container 379ab7aff5480d1cb94642e3a0ed055d95ad1c57be1b08bd095461e5d1a90157: Status 404 returned error can't find the container with id 379ab7aff5480d1cb94642e3a0ed055d95ad1c57be1b08bd095461e5d1a90157 Nov 03 21:46:18.145690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:18.141124 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:18.173700 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:18.153018 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:46:18.173700 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:18.163681 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:18.173700 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:18.167509 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59"] Nov 03 21:46:18.412792 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:18.408771 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 21:46:18.412792 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:18.409412 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:46:18.491695 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:18.486831 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:46:18.614701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:18.599764 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:46:18.614701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:18.599837 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" podUID="ca8a2829-0a62-49af-b6ba-9546aa670350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:18.744631 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:18.739762 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:18.752559 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:18.752517 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:46:18.837202 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:18.837158 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:18.838050 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:18.838025 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:46:18.915756 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:18.912176 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:46:19.096434 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:19.095381 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:46:19.175208 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:19.175112 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:46:19.442032 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:19.441922 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" event={"ID":"ffd0a1d5-d575-4352-b513-130a585dbeec","Type":"ContainerStarted","Data":"4f843bce1feae0f551fe66d8d2fdf55ad9929a640eecf6ccfd53aa1d711cf32c"} Nov 03 21:46:19.442032 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:19.441995 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" event={"ID":"ffd0a1d5-d575-4352-b513-130a585dbeec","Type":"ContainerStarted","Data":"379ab7aff5480d1cb94642e3a0ed055d95ad1c57be1b08bd095461e5d1a90157"} Nov 03 21:46:19.485458 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:19.485402 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" podStartSLOduration=4.485385791 podStartE2EDuration="4.485385791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:19.48424379 +0000 UTC m=+994.991472092" watchObservedRunningTime="2026-11-03 21:46:19.485385791 +0000 UTC m=+994.992614133" Nov 03 21:46:20.452435 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:20.452393 5041 generic.go:356] "Generic (PLEG): container finished" podID="ffd0a1d5-d575-4352-b513-130a585dbeec" containerID="4f843bce1feae0f551fe66d8d2fdf55ad9929a640eecf6ccfd53aa1d711cf32c" exitCode=0 Nov 03 21:46:20.452876 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:20.452475 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" event={"ID":"ffd0a1d5-d575-4352-b513-130a585dbeec","Type":"ContainerDied","Data":"4f843bce1feae0f551fe66d8d2fdf55ad9929a640eecf6ccfd53aa1d711cf32c"} Nov 03 21:46:21.636914 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:21.636883 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" Nov 03 21:46:21.796322 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:21.796232 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd0a1d5-d575-4352-b513-130a585dbeec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd0a1d5-d575-4352-b513-130a585dbeec-maas-api-xks-ca\") pod \"ffd0a1d5-d575-4352-b513-130a585dbeec\" (UID: \"ffd0a1d5-d575-4352-b513-130a585dbeec\") " Nov 03 21:46:21.796484 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:21.796463 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd0a1d5-d575-4352-b513-130a585dbeec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd0a1d5-d575-4352-b513-130a585dbeec-maas-api-openshift-ca\") pod \"ffd0a1d5-d575-4352-b513-130a585dbeec\" (UID: \"ffd0a1d5-d575-4352-b513-130a585dbeec\") " Nov 03 21:46:21.796538 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:21.796505 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd0a1d5-d575-4352-b513-130a585dbeec-kube-api-access-mt5vs\" (UniqueName: \"kubernetes.io/projected/ffd0a1d5-d575-4352-b513-130a585dbeec-kube-api-access-mt5vs\") pod \"ffd0a1d5-d575-4352-b513-130a585dbeec\" (UID: \"ffd0a1d5-d575-4352-b513-130a585dbeec\") " Nov 03 21:46:21.796884 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:21.796853 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd0a1d5-d575-4352-b513-130a585dbeec-maas-api-openshift-ca" pod "ffd0a1d5-d575-4352-b513-130a585dbeec" (UID: "ffd0a1d5-d575-4352-b513-130a585dbeec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:46:21.797199 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:21.797178 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd0a1d5-d575-4352-b513-130a585dbeec-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:21.798795 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:21.798769 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd0a1d5-d575-4352-b513-130a585dbeec-maas-api-xks-ca" pod "ffd0a1d5-d575-4352-b513-130a585dbeec" (UID: "ffd0a1d5-d575-4352-b513-130a585dbeec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:46:21.798885 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:21.798870 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd0a1d5-d575-4352-b513-130a585dbeec-kube-api-access-mt5vs" pod "ffd0a1d5-d575-4352-b513-130a585dbeec" (UID: "ffd0a1d5-d575-4352-b513-130a585dbeec"). InnerVolumeSpecName "kube-api-access-mt5vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:46:21.898514 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:21.898478 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt5vs\" (UniqueName: \"kubernetes.io/projected/ffd0a1d5-d575-4352-b513-130a585dbeec-kube-api-access-mt5vs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:21.898514 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:21.898509 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd0a1d5-d575-4352-b513-130a585dbeec-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:21.952846 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:21.952806 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98"] Nov 03 21:46:21.954237 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:21.954207 5041 cpu_manager.go:414] "containerMap: removing container" podUID="ffd0a1d5-d575-4352-b513-130a585dbeec" containerName="revoke-keys" Nov 03 21:46:21.954415 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:21.954401 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd0a1d5-d575-4352-b513-130a585dbeec" containerName="revoke-keys" Nov 03 21:46:21.954713 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:21.954699 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd0a1d5-d575-4352-b513-130a585dbeec" containerName="revoke-keys" Nov 03 21:46:21.974500 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:21.973651 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" Nov 03 21:46:21.976033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:21.976001 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98"] Nov 03 21:46:22.110039 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:22.109935 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/703148d0-66d5-40ab-844c-6299df569634-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e10523d1-zns98\" (UID: \"703148d0-66d5-40ab-844c-6299df569634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" Nov 03 21:46:22.110223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:22.110063 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/703148d0-66d5-40ab-844c-6299df569634-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e10523d1-zns98\" (UID: \"703148d0-66d5-40ab-844c-6299df569634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" Nov 03 21:46:22.110223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:22.110135 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q87m8\" (UniqueName: \"kubernetes.io/projected/703148d0-66d5-40ab-844c-6299df569634-kube-api-access-q87m8\") pod \"maas-api-revoke-sub-e10523d1-zns98\" (UID: \"703148d0-66d5-40ab-844c-6299df569634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" Nov 03 21:46:22.211429 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:22.211386 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/703148d0-66d5-40ab-844c-6299df569634-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e10523d1-zns98\" (UID: \"703148d0-66d5-40ab-844c-6299df569634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" Nov 03 21:46:22.211603 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:22.211454 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/703148d0-66d5-40ab-844c-6299df569634-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e10523d1-zns98\" (UID: \"703148d0-66d5-40ab-844c-6299df569634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" Nov 03 21:46:22.211603 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:22.211515 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87m8\" (UniqueName: \"kubernetes.io/projected/703148d0-66d5-40ab-844c-6299df569634-kube-api-access-q87m8\") pod \"maas-api-revoke-sub-e10523d1-zns98\" (UID: \"703148d0-66d5-40ab-844c-6299df569634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" Nov 03 21:46:22.212390 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:22.212354 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/703148d0-66d5-40ab-844c-6299df569634-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e10523d1-zns98\" (UID: \"703148d0-66d5-40ab-844c-6299df569634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" Nov 03 21:46:22.213995 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:22.213966 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/703148d0-66d5-40ab-844c-6299df569634-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e10523d1-zns98\" (UID: \"703148d0-66d5-40ab-844c-6299df569634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" Nov 03 21:46:22.235225 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:22.235198 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q87m8\" (UniqueName: \"kubernetes.io/projected/703148d0-66d5-40ab-844c-6299df569634-kube-api-access-q87m8\") pod \"maas-api-revoke-sub-e10523d1-zns98\" (UID: \"703148d0-66d5-40ab-844c-6299df569634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" Nov 03 21:46:22.327447 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:22.327413 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" Nov 03 21:46:22.471729 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:22.471696 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" Nov 03 21:46:22.471917 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:22.471696 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59" event={"ID":"ffd0a1d5-d575-4352-b513-130a585dbeec","Type":"ContainerDied","Data":"379ab7aff5480d1cb94642e3a0ed055d95ad1c57be1b08bd095461e5d1a90157"} Nov 03 21:46:22.471917 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:22.471883 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="379ab7aff5480d1cb94642e3a0ed055d95ad1c57be1b08bd095461e5d1a90157" Nov 03 21:46:22.632333 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:22.632302 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98"] Nov 03 21:46:22.633944 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:46:22.633911 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703148d0_66d5_40ab_844c_6299df569634.slice/crio-06a09fc37ca2503c374bed62e62e1d1dbaca34ed34b6b2ed85d4c576c3f354b5 WatchSource:0}: Error finding container 06a09fc37ca2503c374bed62e62e1d1dbaca34ed34b6b2ed85d4c576c3f354b5: Status 404 returned error can't find the container with id 06a09fc37ca2503c374bed62e62e1d1dbaca34ed34b6b2ed85d4c576c3f354b5 Nov 03 21:46:23.481882 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:23.481844 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" event={"ID":"703148d0-66d5-40ab-844c-6299df569634","Type":"ContainerStarted","Data":"62da3297e8bed3fc3eb32eaefd3b06307fcf9673b8f9c837fd8d834c8b59e2b5"} Nov 03 21:46:23.482266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:23.481906 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" event={"ID":"703148d0-66d5-40ab-844c-6299df569634","Type":"ContainerStarted","Data":"06a09fc37ca2503c374bed62e62e1d1dbaca34ed34b6b2ed85d4c576c3f354b5"} Nov 03 21:46:23.522480 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:23.522420 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" podStartSLOduration=2.522406085 podStartE2EDuration="2.522406085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:23.520167182 +0000 UTC m=+999.027395463" watchObservedRunningTime="2026-11-03 21:46:23.522406085 +0000 UTC m=+999.029634364" Nov 03 21:46:24.492959 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:24.492926 5041 generic.go:356] "Generic (PLEG): container finished" podID="703148d0-66d5-40ab-844c-6299df569634" containerID="62da3297e8bed3fc3eb32eaefd3b06307fcf9673b8f9c837fd8d834c8b59e2b5" exitCode=0 Nov 03 21:46:24.493426 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:24.493005 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" event={"ID":"703148d0-66d5-40ab-844c-6299df569634","Type":"ContainerDied","Data":"62da3297e8bed3fc3eb32eaefd3b06307fcf9673b8f9c837fd8d834c8b59e2b5"} Nov 03 21:46:25.755939 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:25.755910 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" Nov 03 21:46:25.770196 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:25.770160 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/703148d0-66d5-40ab-844c-6299df569634-kube-api-access-q87m8\" (UniqueName: \"kubernetes.io/projected/703148d0-66d5-40ab-844c-6299df569634-kube-api-access-q87m8\") pod \"703148d0-66d5-40ab-844c-6299df569634\" (UID: \"703148d0-66d5-40ab-844c-6299df569634\") " Nov 03 21:46:25.770366 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:25.770213 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/703148d0-66d5-40ab-844c-6299df569634-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/703148d0-66d5-40ab-844c-6299df569634-maas-api-openshift-ca\") pod \"703148d0-66d5-40ab-844c-6299df569634\" (UID: \"703148d0-66d5-40ab-844c-6299df569634\") " Nov 03 21:46:25.770366 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:25.770309 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/703148d0-66d5-40ab-844c-6299df569634-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/703148d0-66d5-40ab-844c-6299df569634-maas-api-xks-ca\") pod \"703148d0-66d5-40ab-844c-6299df569634\" (UID: \"703148d0-66d5-40ab-844c-6299df569634\") " Nov 03 21:46:25.770728 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:25.770651 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/703148d0-66d5-40ab-844c-6299df569634-maas-api-openshift-ca" pod "703148d0-66d5-40ab-844c-6299df569634" (UID: "703148d0-66d5-40ab-844c-6299df569634"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:46:25.770859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:25.770841 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/703148d0-66d5-40ab-844c-6299df569634-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:25.772399 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:25.772366 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/703148d0-66d5-40ab-844c-6299df569634-kube-api-access-q87m8" pod "703148d0-66d5-40ab-844c-6299df569634" (UID: "703148d0-66d5-40ab-844c-6299df569634"). InnerVolumeSpecName "kube-api-access-q87m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:46:25.772567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:25.772545 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/703148d0-66d5-40ab-844c-6299df569634-maas-api-xks-ca" pod "703148d0-66d5-40ab-844c-6299df569634" (UID: "703148d0-66d5-40ab-844c-6299df569634"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:46:25.872923 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:25.872880 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q87m8\" (UniqueName: \"kubernetes.io/projected/703148d0-66d5-40ab-844c-6299df569634-kube-api-access-q87m8\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:25.872923 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:25.872918 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/703148d0-66d5-40ab-844c-6299df569634-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:26.538518 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:26.538486 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" Nov 03 21:46:26.538518 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:26.538502 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98" event={"ID":"703148d0-66d5-40ab-844c-6299df569634","Type":"ContainerDied","Data":"06a09fc37ca2503c374bed62e62e1d1dbaca34ed34b6b2ed85d4c576c3f354b5"} Nov 03 21:46:26.538785 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:26.538552 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06a09fc37ca2503c374bed62e62e1d1dbaca34ed34b6b2ed85d4c576c3f354b5" Nov 03 21:46:28.926093 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:28.926050 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj"] Nov 03 21:46:28.926964 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:28.926941 5041 cpu_manager.go:414] "containerMap: removing container" podUID="703148d0-66d5-40ab-844c-6299df569634" containerName="revoke-keys" Nov 03 21:46:28.926964 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:28.926962 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="703148d0-66d5-40ab-844c-6299df569634" containerName="revoke-keys" Nov 03 21:46:28.927152 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:28.927111 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="703148d0-66d5-40ab-844c-6299df569634" containerName="revoke-keys" Nov 03 21:46:28.932676 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:28.932648 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj" Nov 03 21:46:28.948072 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:28.948046 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj"] Nov 03 21:46:28.955492 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:28.955469 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:46:28.955648 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:28.955626 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:46:29.021576 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:29.021528 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec01e33-6310-4936-8620-117f0ab603eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8811c72-z9vcj\" (UID: \"7ec01e33-6310-4936-8620-117f0ab603eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj" Nov 03 21:46:29.021827 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:29.021750 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec01e33-6310-4936-8620-117f0ab603eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8811c72-z9vcj\" (UID: \"7ec01e33-6310-4936-8620-117f0ab603eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj" Nov 03 21:46:29.021988 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:29.021963 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns4kb\" (UniqueName: \"kubernetes.io/projected/7ec01e33-6310-4936-8620-117f0ab603eb-kube-api-access-ns4kb\") pod \"maas-api-revoke-sub-b8811c72-z9vcj\" (UID: \"7ec01e33-6310-4936-8620-117f0ab603eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj" Nov 03 21:46:29.124066 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:29.124018 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns4kb\" (UniqueName: \"kubernetes.io/projected/7ec01e33-6310-4936-8620-117f0ab603eb-kube-api-access-ns4kb\") pod \"maas-api-revoke-sub-b8811c72-z9vcj\" (UID: \"7ec01e33-6310-4936-8620-117f0ab603eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj" Nov 03 21:46:29.124263 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:29.124146 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec01e33-6310-4936-8620-117f0ab603eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8811c72-z9vcj\" (UID: \"7ec01e33-6310-4936-8620-117f0ab603eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj" Nov 03 21:46:29.125156 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:29.124252 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec01e33-6310-4936-8620-117f0ab603eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8811c72-z9vcj\" (UID: \"7ec01e33-6310-4936-8620-117f0ab603eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj" Nov 03 21:46:29.125156 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:29.125114 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec01e33-6310-4936-8620-117f0ab603eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8811c72-z9vcj\" (UID: \"7ec01e33-6310-4936-8620-117f0ab603eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj" Nov 03 21:46:29.127005 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:29.126976 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec01e33-6310-4936-8620-117f0ab603eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8811c72-z9vcj\" (UID: \"7ec01e33-6310-4936-8620-117f0ab603eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj" Nov 03 21:46:29.145489 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:29.145461 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns4kb\" (UniqueName: \"kubernetes.io/projected/7ec01e33-6310-4936-8620-117f0ab603eb-kube-api-access-ns4kb\") pod \"maas-api-revoke-sub-b8811c72-z9vcj\" (UID: \"7ec01e33-6310-4936-8620-117f0ab603eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj" Nov 03 21:46:29.250526 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:29.250491 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj" Nov 03 21:46:29.555994 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:46:29.555960 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec01e33_6310_4936_8620_117f0ab603eb.slice/crio-459d88a232e4b070277539f2d5fc88519c46035d3a3cee6031358f0e3d747757 WatchSource:0}: Error finding container 459d88a232e4b070277539f2d5fc88519c46035d3a3cee6031358f0e3d747757: Status 404 returned error can't find the container with id 459d88a232e4b070277539f2d5fc88519c46035d3a3cee6031358f0e3d747757 Nov 03 21:46:29.558798 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:29.558264 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj"] Nov 03 21:46:29.569733 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:29.569303 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj" event={"ID":"7ec01e33-6310-4936-8620-117f0ab603eb","Type":"ContainerStarted","Data":"459d88a232e4b070277539f2d5fc88519c46035d3a3cee6031358f0e3d747757"} Nov 03 21:46:30.580264 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:30.580230 5041 generic.go:356] "Generic (PLEG): container finished" podID="7ec01e33-6310-4936-8620-117f0ab603eb" containerID="d1028a2766cedaa9928cd8b849b3290ddf6cd579992b6c24299c497fdd2d8c42" exitCode=0 Nov 03 21:46:30.580789 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:30.580317 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj" event={"ID":"7ec01e33-6310-4936-8620-117f0ab603eb","Type":"ContainerDied","Data":"d1028a2766cedaa9928cd8b849b3290ddf6cd579992b6c24299c497fdd2d8c42"} Nov 03 21:46:31.758810 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:31.758783 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj" Nov 03 21:46:31.870144 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:31.870045 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec01e33-6310-4936-8620-117f0ab603eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec01e33-6310-4936-8620-117f0ab603eb-maas-api-openshift-ca\") pod \"7ec01e33-6310-4936-8620-117f0ab603eb\" (UID: \"7ec01e33-6310-4936-8620-117f0ab603eb\") " Nov 03 21:46:31.870144 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:31.870110 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ec01e33-6310-4936-8620-117f0ab603eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec01e33-6310-4936-8620-117f0ab603eb-maas-api-xks-ca\") pod \"7ec01e33-6310-4936-8620-117f0ab603eb\" (UID: \"7ec01e33-6310-4936-8620-117f0ab603eb\") " Nov 03 21:46:31.870384 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:31.870259 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec01e33-6310-4936-8620-117f0ab603eb-kube-api-access-ns4kb\" (UniqueName: \"kubernetes.io/projected/7ec01e33-6310-4936-8620-117f0ab603eb-kube-api-access-ns4kb\") pod \"7ec01e33-6310-4936-8620-117f0ab603eb\" (UID: \"7ec01e33-6310-4936-8620-117f0ab603eb\") " Nov 03 21:46:31.870477 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:31.870452 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec01e33-6310-4936-8620-117f0ab603eb-maas-api-openshift-ca" pod "7ec01e33-6310-4936-8620-117f0ab603eb" (UID: "7ec01e33-6310-4936-8620-117f0ab603eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:46:31.870812 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:31.870793 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec01e33-6310-4936-8620-117f0ab603eb-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:31.872319 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:31.872292 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec01e33-6310-4936-8620-117f0ab603eb-kube-api-access-ns4kb" pod "7ec01e33-6310-4936-8620-117f0ab603eb" (UID: "7ec01e33-6310-4936-8620-117f0ab603eb"). InnerVolumeSpecName "kube-api-access-ns4kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:46:31.872319 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:31.872302 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec01e33-6310-4936-8620-117f0ab603eb-maas-api-xks-ca" pod "7ec01e33-6310-4936-8620-117f0ab603eb" (UID: "7ec01e33-6310-4936-8620-117f0ab603eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:46:31.972797 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:31.972762 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec01e33-6310-4936-8620-117f0ab603eb-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:31.972797 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:31.972799 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns4kb\" (UniqueName: \"kubernetes.io/projected/7ec01e33-6310-4936-8620-117f0ab603eb-kube-api-access-ns4kb\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:32.599509 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:32.599464 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj" Nov 03 21:46:32.599509 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:32.599496 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj" event={"ID":"7ec01e33-6310-4936-8620-117f0ab603eb","Type":"ContainerDied","Data":"459d88a232e4b070277539f2d5fc88519c46035d3a3cee6031358f0e3d747757"} Nov 03 21:46:32.599824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:32.599546 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="459d88a232e4b070277539f2d5fc88519c46035d3a3cee6031358f0e3d747757" Nov 03 21:46:34.977762 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:34.977723 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc"] Nov 03 21:46:34.978442 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:34.978427 5041 cpu_manager.go:414] "containerMap: removing container" podUID="7ec01e33-6310-4936-8620-117f0ab603eb" containerName="revoke-keys" Nov 03 21:46:34.978487 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:34.978443 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec01e33-6310-4936-8620-117f0ab603eb" containerName="revoke-keys" Nov 03 21:46:34.978586 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:34.978576 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec01e33-6310-4936-8620-117f0ab603eb" containerName="revoke-keys" Nov 03 21:46:34.982099 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:34.982074 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.037263 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.037219 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc"] Nov 03 21:46:35.121819 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.121777 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-workload-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.121819 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.121824 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-istio-data\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.122096 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.121921 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-credential-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.122096 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.121963 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6js8r\" (UniqueName: \"kubernetes.io/projected/b68bf331-de7c-4e66-9371-72984471e65e-kube-api-access-6js8r\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.122096 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.122017 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68bf331-de7c-4e66-9371-72984471e65e-istio-token\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.122252 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.122179 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-istio-envoy\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.122379 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.122360 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-workload-certs\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.122442 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.122390 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68bf331-de7c-4e66-9371-72984471e65e-istiod-ca-cert\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.122499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.122485 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68bf331-de7c-4e66-9371-72984471e65e-istio-podinfo\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.224269 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.224231 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-workload-certs\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.224269 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.224271 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68bf331-de7c-4e66-9371-72984471e65e-istiod-ca-cert\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.224560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.224374 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68bf331-de7c-4e66-9371-72984471e65e-istio-podinfo\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.224560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.224463 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-workload-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.224560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.224494 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-istio-data\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.224560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.224550 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-credential-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.224837 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.224616 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6js8r\" (UniqueName: \"kubernetes.io/projected/b68bf331-de7c-4e66-9371-72984471e65e-kube-api-access-6js8r\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.224837 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.224788 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-workload-certs\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.224947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.224831 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-workload-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.225046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.225021 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-credential-socket\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.225108 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.225048 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-istio-data\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.225108 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.225062 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68bf331-de7c-4e66-9371-72984471e65e-istiod-ca-cert\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.225217 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.225112 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68bf331-de7c-4e66-9371-72984471e65e-istio-token\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.225322 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.225298 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-istio-envoy\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.227435 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.227404 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-istio-envoy\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.227730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.227711 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68bf331-de7c-4e66-9371-72984471e65e-istio-podinfo\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.246366 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.246283 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68bf331-de7c-4e66-9371-72984471e65e-istio-token\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.246537 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.246377 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6js8r\" (UniqueName: \"kubernetes.io/projected/b68bf331-de7c-4e66-9371-72984471e65e-kube-api-access-6js8r\") pod \"e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.297070 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.297036 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:35.634277 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.634242 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc"] Nov 03 21:46:35.638826 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.638789 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:35.638947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.638863 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:35.638947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.638925 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:35.892856 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.892753 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf"] Nov 03 21:46:35.898932 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.898904 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:35.917955 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.917918 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf"] Nov 03 21:46:35.929696 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.929192 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-05bc27\"/\"openshift-service-ca.crt\"" Nov 03 21:46:35.932825 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.930049 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-05bc27\"/\"default-dockercfg-dc6k9\"" Nov 03 21:46:35.932825 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.930147 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-05bc27\"/\"distinct-2-w4-05bc27-kserve-self-signed-certs\"" Nov 03 21:46:35.932825 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:35.930227 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-05bc27\"/\"kube-root-ca.crt\"" Nov 03 21:46:36.041509 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.041466 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-tmp-dir\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.042144 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.042113 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-model-cache\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.042257 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.042168 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-home\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.042388 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.042361 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-dshm\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.042510 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.042492 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf2qp\" (UniqueName: \"kubernetes.io/projected/3f52ca60-4d7e-4707-874a-02632382a76d-kube-api-access-vf2qp\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.042629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.042605 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f52ca60-4d7e-4707-874a-02632382a76d-tls-certs\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.144250 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.144142 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-tmp-dir\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.144446 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.144250 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-model-cache\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.144446 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.144301 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-home\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.144446 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.144441 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-dshm\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.144624 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.144538 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf2qp\" (UniqueName: \"kubernetes.io/projected/3f52ca60-4d7e-4707-874a-02632382a76d-kube-api-access-vf2qp\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.144696 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.144630 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f52ca60-4d7e-4707-874a-02632382a76d-tls-certs\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.145208 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.145182 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-tmp-dir\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.146291 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.145633 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-model-cache\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.146463 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.145745 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-home\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.150246 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.150084 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f52ca60-4d7e-4707-874a-02632382a76d-tls-certs\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.154932 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.154900 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-dshm\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.169964 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.169926 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf2qp\" (UniqueName: \"kubernetes.io/projected/3f52ca60-4d7e-4707-874a-02632382a76d-kube-api-access-vf2qp\") pod \"distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.244078 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.244040 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:36.270165 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.270127 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh"] Nov 03 21:46:36.276176 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.276144 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" Nov 03 21:46:36.292844 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.292812 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:46:36.293000 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.292887 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:46:36.321782 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.320577 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh"] Nov 03 21:46:36.475650 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.475606 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvt2v\" (UniqueName: \"kubernetes.io/projected/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-kube-api-access-pvt2v\") pod \"maas-api-revoke-sub-da63b65f-bmlgh\" (UID: \"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" Nov 03 21:46:36.475650 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.475652 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da63b65f-bmlgh\" (UID: \"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" Nov 03 21:46:36.475958 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.475899 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da63b65f-bmlgh\" (UID: \"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" Nov 03 21:46:36.577269 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.577229 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da63b65f-bmlgh\" (UID: \"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" Nov 03 21:46:36.577445 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.577394 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvt2v\" (UniqueName: \"kubernetes.io/projected/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-kube-api-access-pvt2v\") pod \"maas-api-revoke-sub-da63b65f-bmlgh\" (UID: \"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" Nov 03 21:46:36.577445 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.577416 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da63b65f-bmlgh\" (UID: \"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" Nov 03 21:46:36.578150 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.578125 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da63b65f-bmlgh\" (UID: \"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" Nov 03 21:46:36.579764 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.579705 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da63b65f-bmlgh\" (UID: \"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" Nov 03 21:46:36.601524 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.601492 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvt2v\" (UniqueName: \"kubernetes.io/projected/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-kube-api-access-pvt2v\") pod \"maas-api-revoke-sub-da63b65f-bmlgh\" (UID: \"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" Nov 03 21:46:36.603399 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:46:36.603369 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f52ca60_4d7e_4707_874a_02632382a76d.slice/crio-4df39328d08e0b0abaeac3d51c2ab5d9b4311cd865b161f087bbcf825540f454 WatchSource:0}: Error finding container 4df39328d08e0b0abaeac3d51c2ab5d9b4311cd865b161f087bbcf825540f454: Status 404 returned error can't find the container with id 4df39328d08e0b0abaeac3d51c2ab5d9b4311cd865b161f087bbcf825540f454 Nov 03 21:46:36.605236 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.605214 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf"] Nov 03 21:46:36.654029 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.653986 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" event={"ID":"3f52ca60-4d7e-4707-874a-02632382a76d","Type":"ContainerStarted","Data":"4df39328d08e0b0abaeac3d51c2ab5d9b4311cd865b161f087bbcf825540f454"} Nov 03 21:46:36.656310 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.656275 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" event={"ID":"b68bf331-de7c-4e66-9371-72984471e65e","Type":"ContainerStarted","Data":"8377c76c2c5611506b0db9607b8ecb564ec0325f748ef853b32baeb6f123852b"} Nov 03 21:46:36.656448 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.656336 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" event={"ID":"b68bf331-de7c-4e66-9371-72984471e65e","Type":"ContainerStarted","Data":"fe0e90c50c255fe633e1aae2b5fc636426cacd15a4a08a4ed8d82199cb8e105f"} Nov 03 21:46:36.672841 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.672809 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" Nov 03 21:46:36.703524 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:36.703475 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" podStartSLOduration=2.703458007 podStartE2EDuration="2.703458007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:36.700199099 +0000 UTC m=+1012.207427418" watchObservedRunningTime="2026-11-03 21:46:36.703458007 +0000 UTC m=+1012.210686287" Nov 03 21:46:37.044835 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:37.044727 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh"] Nov 03 21:46:37.049919 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:46:37.049884 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafa0ffa_bbbf_4a07_a651_6cbc1ff38939.slice/crio-c6b3db2f8a6cfb1d504e487c6113724522ee055668a9d8d162821120c9b7cd16 WatchSource:0}: Error finding container c6b3db2f8a6cfb1d504e487c6113724522ee055668a9d8d162821120c9b7cd16: Status 404 returned error can't find the container with id c6b3db2f8a6cfb1d504e487c6113724522ee055668a9d8d162821120c9b7cd16 Nov 03 21:46:37.298207 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:37.298123 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:37.299775 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:37.299748 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 21:46:37.299933 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:37.299804 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" podUID="b68bf331-de7c-4e66-9371-72984471e65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 21:46:37.668956 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:37.668863 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" event={"ID":"3f52ca60-4d7e-4707-874a-02632382a76d","Type":"ContainerStarted","Data":"d2587b4dd0fd47ef23d48b30df1bf0a07d494e9451beb3fa77b73fa71aa1ed35"} Nov 03 21:46:37.670698 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:37.670654 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" event={"ID":"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939","Type":"ContainerStarted","Data":"246b0103072bd417285fa7be281166b859deb7792cfd9b556a563cf82917bdf8"} Nov 03 21:46:37.670832 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:37.670716 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" event={"ID":"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939","Type":"ContainerStarted","Data":"c6b3db2f8a6cfb1d504e487c6113724522ee055668a9d8d162821120c9b7cd16"} Nov 03 21:46:37.744688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:37.743056 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" podStartSLOduration=2.743033942 podStartE2EDuration="2.743033942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:37.719145314 +0000 UTC m=+1013.226373595" watchObservedRunningTime="2026-11-03 21:46:37.743033942 +0000 UTC m=+1013.250262224" Nov 03 21:46:37.770240 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:37.770184 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" podStartSLOduration=1.770169179 podStartE2EDuration="1.770169179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:37.764430113 +0000 UTC m=+1013.271658392" watchObservedRunningTime="2026-11-03 21:46:37.770169179 +0000 UTC m=+1013.277397481" Nov 03 21:46:38.248835 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:38.248805 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:38.298103 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:38.298070 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 21:46:38.298265 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:38.298126 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" podUID="b68bf331-de7c-4e66-9371-72984471e65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 21:46:38.681077 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:38.680990 5041 generic.go:356] "Generic (PLEG): container finished" podID="fafa0ffa-bbbf-4a07-a651-6cbc1ff38939" containerID="246b0103072bd417285fa7be281166b859deb7792cfd9b556a563cf82917bdf8" exitCode=0 Nov 03 21:46:38.681259 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:38.681074 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" event={"ID":"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939","Type":"ContainerDied","Data":"246b0103072bd417285fa7be281166b859deb7792cfd9b556a563cf82917bdf8"} Nov 03 21:46:39.298322 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:39.298283 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 21:46:39.298731 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:39.298347 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" podUID="b68bf331-de7c-4e66-9371-72984471e65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 21:46:40.006297 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.006246 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" Nov 03 21:46:40.058748 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.058715 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs"] Nov 03 21:46:40.060070 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.060044 5041 cpu_manager.go:414] "containerMap: removing container" podUID="fafa0ffa-bbbf-4a07-a651-6cbc1ff38939" containerName="revoke-keys" Nov 03 21:46:40.060209 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.060198 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fafa0ffa-bbbf-4a07-a651-6cbc1ff38939" containerName="revoke-keys" Nov 03 21:46:40.060456 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.060444 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fafa0ffa-bbbf-4a07-a651-6cbc1ff38939" containerName="revoke-keys" Nov 03 21:46:40.064479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.064464 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.068145 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.067016 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn"] Nov 03 21:46:40.072707 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.072685 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-maas-api-openshift-ca\") pod \"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939\" (UID: \"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939\") " Nov 03 21:46:40.072916 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.072899 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-kube-api-access-pvt2v\" (UniqueName: \"kubernetes.io/projected/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-kube-api-access-pvt2v\") pod \"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939\" (UID: \"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939\") " Nov 03 21:46:40.073137 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.073099 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-maas-api-xks-ca\") pod \"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939\" (UID: \"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939\") " Nov 03 21:46:40.074618 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.074596 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-maas-api-openshift-ca" pod "fafa0ffa-bbbf-4a07-a651-6cbc1ff38939" (UID: "fafa0ffa-bbbf-4a07-a651-6cbc1ff38939"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:46:40.084452 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.084009 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.099003 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.098972 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-maas-api-xks-ca" pod "fafa0ffa-bbbf-4a07-a651-6cbc1ff38939" (UID: "fafa0ffa-bbbf-4a07-a651-6cbc1ff38939"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:46:40.104787 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.104757 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-kube-api-access-pvt2v" pod "fafa0ffa-bbbf-4a07-a651-6cbc1ff38939" (UID: "fafa0ffa-bbbf-4a07-a651-6cbc1ff38939"). InnerVolumeSpecName "kube-api-access-pvt2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:46:40.158285 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.158208 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs"] Nov 03 21:46:40.164643 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.164612 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn"] Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.175910 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-credential-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.175965 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49882dd3-115a-4449-b98d-9773358608bc-istio-token\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.175999 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-workload-certs\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176055 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-istio-data\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176101 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-workload-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176158 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49882dd3-115a-4449-b98d-9773358608bc-istiod-ca-cert\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176190 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-workload-certs\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176258 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-token\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176292 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4mv9\" (UniqueName: \"kubernetes.io/projected/49882dd3-115a-4449-b98d-9773358608bc-kube-api-access-s4mv9\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176331 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-istio-envoy\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176362 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367832c5-176f-4416-afd0-5a1bdb2b7570-istiod-ca-cert\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176435 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w794m\" (UniqueName: \"kubernetes.io/projected/367832c5-176f-4416-afd0-5a1bdb2b7570-kube-api-access-w794m\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176588 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-credential-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176621 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-data\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176674 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-workload-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176737 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49882dd3-115a-4449-b98d-9773358608bc-istio-podinfo\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176763 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-envoy\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176786 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-podinfo\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176888 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176905 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvt2v\" (UniqueName: \"kubernetes.io/projected/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-kube-api-access-pvt2v\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:40.178645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.176920 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:40.277731 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.277634 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367832c5-176f-4416-afd0-5a1bdb2b7570-istiod-ca-cert\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.277905 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.277729 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w794m\" (UniqueName: \"kubernetes.io/projected/367832c5-176f-4416-afd0-5a1bdb2b7570-kube-api-access-w794m\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.277905 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.277838 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-credential-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.277905 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.277861 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-data\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.277905 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.277886 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-workload-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.278128 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.277956 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49882dd3-115a-4449-b98d-9773358608bc-istio-podinfo\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.278128 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.277981 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-envoy\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.278128 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.278114 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-podinfo\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.278423 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.278390 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-workload-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.278495 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.278462 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367832c5-176f-4416-afd0-5a1bdb2b7570-istiod-ca-cert\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.278546 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.278477 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-credential-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.278546 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.278511 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-credential-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.278646 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.278541 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49882dd3-115a-4449-b98d-9773358608bc-istio-token\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.278646 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.278568 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-workload-certs\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.278646 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.278621 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-istio-data\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.278828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.278699 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-workload-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.278828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.278757 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49882dd3-115a-4449-b98d-9773358608bc-istiod-ca-cert\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.278828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.278775 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-credential-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.278828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.278789 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-workload-certs\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.279068 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.278877 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-token\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.279068 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.278915 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4mv9\" (UniqueName: \"kubernetes.io/projected/49882dd3-115a-4449-b98d-9773358608bc-kube-api-access-s4mv9\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.279068 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.278929 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-data\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.279258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.279064 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-workload-certs\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.279258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.279107 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-istio-envoy\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.279258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.279240 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-workload-certs\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.279411 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.279374 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-workload-socket\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.279462 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.279412 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-istio-data\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.280035 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.280005 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49882dd3-115a-4449-b98d-9773358608bc-istiod-ca-cert\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.281918 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.281896 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-envoy\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.282143 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.282118 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-istio-envoy\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.283126 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.283100 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-podinfo\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.283226 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.283100 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49882dd3-115a-4449-b98d-9773358608bc-istio-podinfo\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.297307 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.297281 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 21:46:40.297414 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.297343 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" podUID="b68bf331-de7c-4e66-9371-72984471e65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 21:46:40.300123 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.300101 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-token\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.300603 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.300584 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4mv9\" (UniqueName: \"kubernetes.io/projected/49882dd3-115a-4449-b98d-9773358608bc-kube-api-access-s4mv9\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.300696 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.300614 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w794m\" (UniqueName: \"kubernetes.io/projected/367832c5-176f-4416-afd0-5a1bdb2b7570-kube-api-access-w794m\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.301972 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.301953 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49882dd3-115a-4449-b98d-9773358608bc-istio-token\") pod \"e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.335683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.334866 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr"] Nov 03 21:46:40.349162 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.349124 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.379247 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.378407 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr"] Nov 03 21:46:40.386604 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.386574 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd"] Nov 03 21:46:40.393989 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.393966 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd"] Nov 03 21:46:40.394154 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.394055 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.406955 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.406932 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:40.458416 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.458381 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:40.482778 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.482432 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/149765b9-f8c8-4896-9b15-79f567e82336-istiod-ca-cert\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.482778 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.482573 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-envoy\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.482778 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.482635 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-workload-certs\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.483033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.482803 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhxsv\" (UniqueName: \"kubernetes.io/projected/149765b9-f8c8-4896-9b15-79f567e82336-kube-api-access-bhxsv\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.483033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.482862 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-istio-envoy\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.483033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.482923 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-podinfo\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.483033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.482958 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96g8z\" (UniqueName: \"kubernetes.io/projected/fcda3325-5e04-4611-9a1e-48075b67d03b-kube-api-access-96g8z\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.483033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.482985 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-token\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.483033 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.483026 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/149765b9-f8c8-4896-9b15-79f567e82336-istio-podinfo\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.483259 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.483107 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-istio-data\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.483259 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.483152 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcda3325-5e04-4611-9a1e-48075b67d03b-istiod-ca-cert\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.483259 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.483167 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-credential-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.483259 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.483184 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/149765b9-f8c8-4896-9b15-79f567e82336-istio-token\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.483259 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.483203 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-data\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.483456 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.483368 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-workload-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.483456 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.483413 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-credential-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.483534 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.483506 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-workload-certs\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.483576 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.483540 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-workload-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.584965 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-istio-data\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585029 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcda3325-5e04-4611-9a1e-48075b67d03b-istiod-ca-cert\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585050 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-credential-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585068 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/149765b9-f8c8-4896-9b15-79f567e82336-istio-token\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585085 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-data\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585128 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-workload-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585152 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-credential-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585193 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-workload-certs\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585216 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-workload-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585253 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/149765b9-f8c8-4896-9b15-79f567e82336-istiod-ca-cert\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585329 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-envoy\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585402 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-workload-certs\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585475 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhxsv\" (UniqueName: \"kubernetes.io/projected/149765b9-f8c8-4896-9b15-79f567e82336-kube-api-access-bhxsv\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585531 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-istio-envoy\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585597 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-podinfo\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585641 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96g8z\" (UniqueName: \"kubernetes.io/projected/fcda3325-5e04-4611-9a1e-48075b67d03b-kube-api-access-96g8z\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585686 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-token\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.585714 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/149765b9-f8c8-4896-9b15-79f567e82336-istio-podinfo\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.586567 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-workload-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.586898 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-istio-data\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.587454 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-data\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.587538 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcda3325-5e04-4611-9a1e-48075b67d03b-istiod-ca-cert\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.587773 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-credential-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.587819 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-workload-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.588019 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-credential-socket\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.588087 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-workload-certs\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.588552 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-workload-certs\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.588770 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/149765b9-f8c8-4896-9b15-79f567e82336-istiod-ca-cert\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.591682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.588864 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/149765b9-f8c8-4896-9b15-79f567e82336-istio-podinfo\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.593900 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.591737 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-podinfo\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.593900 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.591961 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-istio-envoy\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.594945 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.594269 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-envoy\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.633827 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.621443 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96g8z\" (UniqueName: \"kubernetes.io/projected/fcda3325-5e04-4611-9a1e-48075b67d03b-kube-api-access-96g8z\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.646457 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.646295 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhxsv\" (UniqueName: \"kubernetes.io/projected/149765b9-f8c8-4896-9b15-79f567e82336-kube-api-access-bhxsv\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.646644 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.646621 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-token\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.647204 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.647181 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/149765b9-f8c8-4896-9b15-79f567e82336-istio-token\") pod \"e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.693277 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.693245 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:40.703448 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.703417 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" Nov 03 21:46:40.704326 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.704297 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh" event={"ID":"fafa0ffa-bbbf-4a07-a651-6cbc1ff38939","Type":"ContainerDied","Data":"c6b3db2f8a6cfb1d504e487c6113724522ee055668a9d8d162821120c9b7cd16"} Nov 03 21:46:40.704476 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.704355 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6b3db2f8a6cfb1d504e487c6113724522ee055668a9d8d162821120c9b7cd16" Nov 03 21:46:40.747982 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.747920 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:40.803750 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.801567 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs"] Nov 03 21:46:40.849720 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:46:40.833029 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:46:40.884479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.878934 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:40.884479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.879118 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:40.884479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.879249 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:40.884479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.880155 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:40.884479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.880281 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:40.884479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.880332 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn"] Nov 03 21:46:40.884479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:40.881996 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:41.174175 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.174141 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr"] Nov 03 21:46:41.182884 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.182518 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:41.182884 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.182603 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:41.182884 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.182640 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:41.282362 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.272868 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd"] Nov 03 21:46:41.282708 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.282648 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:41.282770 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.282754 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:41.282830 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.282799 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:41.297492 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.297456 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 21:46:41.297650 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.297514 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" podUID="b68bf331-de7c-4e66-9371-72984471e65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 21:46:41.816695 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.816030 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" event={"ID":"fcda3325-5e04-4611-9a1e-48075b67d03b","Type":"ContainerStarted","Data":"ec0ccebca5738af3440a181f5aa49e75a6f81bcd757ee90a6a582d02360c2a67"} Nov 03 21:46:41.816695 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.816123 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" event={"ID":"fcda3325-5e04-4611-9a1e-48075b67d03b","Type":"ContainerStarted","Data":"50a47dbab70c3880188b0738e5b5e40a39a0ecc51ea00d44136079be31febd46"} Nov 03 21:46:41.823694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.820094 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" event={"ID":"149765b9-f8c8-4896-9b15-79f567e82336","Type":"ContainerStarted","Data":"711f15b5b8f73ec510f5dda14ead4b1df6b852205a3442ae8b0609cc761678ee"} Nov 03 21:46:41.823694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.820154 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" event={"ID":"149765b9-f8c8-4896-9b15-79f567e82336","Type":"ContainerStarted","Data":"2e1719dafd31ea9b78ecb4e3cbac83ec13590eb7c45581222fd933d592934153"} Nov 03 21:46:41.853566 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.852511 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" event={"ID":"49882dd3-115a-4449-b98d-9773358608bc","Type":"ContainerStarted","Data":"d817f1837d733f59ff5a94fc2062e5a0b8e66c325c3c44d983b2734062e5c71a"} Nov 03 21:46:41.853566 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.852592 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" event={"ID":"49882dd3-115a-4449-b98d-9773358608bc","Type":"ContainerStarted","Data":"c23ea08d486ae9e0a6c42a1cace4450c03684af236d4a4fdf8c4cd82e9e81a23"} Nov 03 21:46:41.903536 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.903494 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" event={"ID":"367832c5-176f-4416-afd0-5a1bdb2b7570","Type":"ContainerStarted","Data":"ce79be379e644fad8ba3c9f4ce0b02c78ab6876da16be67adedb10dad3a219db"} Nov 03 21:46:41.909967 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.909927 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" event={"ID":"367832c5-176f-4416-afd0-5a1bdb2b7570","Type":"ContainerStarted","Data":"55b5a9e1c0fe674eda0e4d014f7c8a388804773777fe5e804269b8f5b49701ce"} Nov 03 21:46:41.950688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:41.931454 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podStartSLOduration=1.931431222 podStartE2EDuration="1.931431222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:41.902999623 +0000 UTC m=+1017.410227930" watchObservedRunningTime="2026-11-03 21:46:41.931431222 +0000 UTC m=+1017.438659505" Nov 03 21:46:42.124464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.124332 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podStartSLOduration=2.124308692 podStartE2EDuration="2.124308692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:42.022689127 +0000 UTC m=+1017.529917384" watchObservedRunningTime="2026-11-03 21:46:42.124308692 +0000 UTC m=+1017.631536972" Nov 03 21:46:42.310694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.304850 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 21:46:42.310694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.304913 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" podUID="b68bf331-de7c-4e66-9371-72984471e65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 21:46:42.419814 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.419748 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:42.428328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.428288 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:46:42.428607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.428578 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 21:46:42.448684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.436057 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podStartSLOduration=2.436031966 podStartE2EDuration="2.436031966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:42.071769961 +0000 UTC m=+1017.578998270" watchObservedRunningTime="2026-11-03 21:46:42.436031966 +0000 UTC m=+1017.943260246" Nov 03 21:46:42.466115 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.466078 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:42.486687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.483465 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podStartSLOduration=2.483442246 podStartE2EDuration="2.483442246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:42.225579409 +0000 UTC m=+1017.732807700" watchObservedRunningTime="2026-11-03 21:46:42.483442246 +0000 UTC m=+1017.990670526" Nov 03 21:46:42.508816 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.486029 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:46:42.509407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.509099 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 21:46:42.709559 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.709523 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:46:42.709913 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.709882 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 21:46:42.710959 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.710930 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:42.756766 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.756728 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:42.763840 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.760820 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:46:42.763840 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.760884 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 21:46:42.829688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:42.825621 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:43.047750 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:43.047190 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:43.052220 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:43.051820 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:43.300761 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:43.300357 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 21:46:43.300761 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:43.300422 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" podUID="b68bf331-de7c-4e66-9371-72984471e65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 21:46:43.409142 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:43.409097 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:46:43.409472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:43.409437 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 21:46:43.475688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:43.462324 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:46:43.475688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:43.462389 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 21:46:43.489812 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:43.489760 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:44.134337 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:44.134286 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:44.205339 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:44.142090 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:46:44.722842 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:44.722787 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 21:46:44.747692 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:44.223748 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 21:46:44.894700 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:44.875071 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" podUID="fa892a3f-4760-45ff-b00b-0840970540bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:45.047840 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:44.375249 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 21:46:45.047840 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:44.505952 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:45.047840 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:44.250955 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:46:45.047840 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:45.045837 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 21:46:45.047840 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:44.917498 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:46:45.050503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:44.251110 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:46:45.076692 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:45.073925 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 21:46:45.076692 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:44.730474 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:46:45.076692 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:45.074054 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 21:46:45.196686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:45.196186 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 03 21:46:45.303889 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:45.284042 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:45.383939 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:45.383889 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 03 21:46:45.620747 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:45.595277 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:46:45.621069 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:45.621033 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 21:46:45.621212 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:44.224355 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:46:45.621862 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:45.621829 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:46:45.736688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:45.675769 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bqstk" podUID="2f3d92ef-2874-44da-90a5-ff115f8c2755" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:45.736688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:45.675950 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:46:45.736688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:45.675985 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 21:46:45.884752 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:45.880775 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:46.152783 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.044906 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 21:46:46.153171 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.046052 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:46.153338 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.048016 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:46.153444 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.056584 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 21:46:46.153553 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.111830 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:46:46.153717 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.153681 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 21:46:46.153845 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.111881 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:46:46.153975 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.153954 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 21:46:46.191656 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.191617 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:46:46.200794 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.200743 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:46.328724 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.312810 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:46:46.328724 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.320750 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 21:46:46.375840 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.375792 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:46.395861 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.395505 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:46:46.395861 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.395580 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 21:46:46.515570 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.515516 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:46:46.515881 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.515837 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" podUID="b68bf331-de7c-4e66-9371-72984471e65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:46.599047 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.599008 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:46:46.599293 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.599263 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 21:46:46.599483 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.599469 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:46:46.599589 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.599574 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 21:46:46.600186 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.600160 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:46:46.600374 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.600355 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 21:46:46.861736 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.861006 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:46:46.861736 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:46.861074 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 21:46:47.279688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.243953 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:47.330875 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.330818 5041 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.120:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:47.497688 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:46:47.485484 5041 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.294s" Nov 03 21:46:47.530078 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.529984 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:46:47.530351 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.530313 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 21:46:47.544694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.531011 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:46:47.648574 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.577086 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 21:46:47.663994 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.663949 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:46:47.664267 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.664233 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" podUID="b68bf331-de7c-4e66-9371-72984471e65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:47.739690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.724369 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:46:47.774453 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.774405 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 21:46:47.774727 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.724844 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:46:47.774864 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.774844 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 21:46:47.802046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.790278 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:46:47.802046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.790335 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 21:46:47.865654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.834160 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:47.930814 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.930777 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh"] Nov 03 21:46:47.983210 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.892197 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:47.983559 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:47.983523 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/health\": context deadline exceeded" Nov 03 21:46:48.009690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.001275 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:48.049767 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.025880 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 21:46:48.089345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.038166 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:46:48.094750 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.038207 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 21:46:48.094990 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.038235 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 21:46:48.134923 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.050228 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:46:48.134923 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.051490 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 21:46:48.255711 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.254030 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:48.261097 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.261062 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh"] Nov 03 21:46:48.261381 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.261364 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:46:48.266803 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.266772 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:46:48.267037 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.267022 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:46:48.267165 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.267154 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:46:48.284790 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.284754 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:46:48.314835 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.314800 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:46:48.356750 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.335880 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh" Nov 03 21:46:48.384100 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.384053 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:46:48.517586 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.476125 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:46:48.594313 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.572483 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 21:46:48.623019 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.538228 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:46:48.655077 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.647225 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/454e2120-320f-4213-b2f5-0c0034a5f05d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dceeffb-hwdxh\" (UID: \"454e2120-320f-4213-b2f5-0c0034a5f05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh" Nov 03 21:46:48.750714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.739451 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/454e2120-320f-4213-b2f5-0c0034a5f05d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dceeffb-hwdxh\" (UID: \"454e2120-320f-4213-b2f5-0c0034a5f05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh" Nov 03 21:46:48.750714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.682458 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:46:48.750714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.491333 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:46:48.750714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.739769 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 21:46:48.750714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.740144 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89bpg\" (UniqueName: \"kubernetes.io/projected/454e2120-320f-4213-b2f5-0c0034a5f05d-kube-api-access-89bpg\") pod \"maas-api-revoke-sub-5dceeffb-hwdxh\" (UID: \"454e2120-320f-4213-b2f5-0c0034a5f05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh" Nov 03 21:46:48.750714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.687770 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:46:48.750714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.740268 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" podUID="b68bf331-de7c-4e66-9371-72984471e65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:48.780217 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.712605 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:46:48.780495 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.780460 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 21:46:48.792802 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.788204 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:46:48.792802 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.788273 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 21:46:48.851471 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.851262 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89bpg\" (UniqueName: \"kubernetes.io/projected/454e2120-320f-4213-b2f5-0c0034a5f05d-kube-api-access-89bpg\") pod \"maas-api-revoke-sub-5dceeffb-hwdxh\" (UID: \"454e2120-320f-4213-b2f5-0c0034a5f05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh" Nov 03 21:46:48.879784 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.879687 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/454e2120-320f-4213-b2f5-0c0034a5f05d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dceeffb-hwdxh\" (UID: \"454e2120-320f-4213-b2f5-0c0034a5f05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh" Nov 03 21:46:48.880863 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.880837 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/454e2120-320f-4213-b2f5-0c0034a5f05d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dceeffb-hwdxh\" (UID: \"454e2120-320f-4213-b2f5-0c0034a5f05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh" Nov 03 21:46:48.901133 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:48.901097 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/454e2120-320f-4213-b2f5-0c0034a5f05d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dceeffb-hwdxh\" (UID: \"454e2120-320f-4213-b2f5-0c0034a5f05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh" Nov 03 21:46:49.113691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.083895 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 21:46:49.113691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.084783 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/454e2120-320f-4213-b2f5-0c0034a5f05d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dceeffb-hwdxh\" (UID: \"454e2120-320f-4213-b2f5-0c0034a5f05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh" Nov 03 21:46:49.130743 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.119029 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:49.130743 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.120349 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:49.158686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.151604 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:46:49.199799 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.185215 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:49.261240 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.261202 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:49.504077 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.472101 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:46:49.505174 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.505127 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 21:46:49.526689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.505597 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:46:49.526689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.512290 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:46:49.526689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.512335 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:49.604002 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.603961 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99"] Nov 03 21:46:49.794734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.794573 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89bpg\" (UniqueName: \"kubernetes.io/projected/454e2120-320f-4213-b2f5-0c0034a5f05d-kube-api-access-89bpg\") pod \"maas-api-revoke-sub-5dceeffb-hwdxh\" (UID: \"454e2120-320f-4213-b2f5-0c0034a5f05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh" Nov 03 21:46:49.897455 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.897416 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:46:49.897798 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.897757 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 21:46:49.900385 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.900356 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:46:49.900571 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.900541 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 21:46:49.921900 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.921860 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:49.966149 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.966101 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:46:49.966412 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.966384 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 21:46:50.010009 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:49.998582 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh" Nov 03 21:46:50.110910 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.089936 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:50.129206 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.129164 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:46:50.251683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.226813 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:50.292026 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.253950 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 21:46:50.292763 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.254006 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 21:46:50.293339 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.290967 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80847094-3884-43b5-8145-78f14994e864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a99bb5-qcw99\" (UID: \"80847094-3884-43b5-8145-78f14994e864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" Nov 03 21:46:50.293613 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.293590 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80847094-3884-43b5-8145-78f14994e864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a99bb5-qcw99\" (UID: \"80847094-3884-43b5-8145-78f14994e864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" Nov 03 21:46:50.293792 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.293773 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx7wb\" (UniqueName: \"kubernetes.io/projected/80847094-3884-43b5-8145-78f14994e864-kube-api-access-cx7wb\") pod \"maas-api-revoke-sub-13a99bb5-qcw99\" (UID: \"80847094-3884-43b5-8145-78f14994e864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" Nov 03 21:46:50.444749 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.304984 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:50.444749 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.305617 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Nov 03 21:46:50.444749 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.305988 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:46:50.444749 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.306201 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" Nov 03 21:46:50.506693 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.498484 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99"] Nov 03 21:46:50.515893 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.515836 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Nov 03 21:46:50.583377 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.583314 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:50.838147 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.838098 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:50.898625 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.898583 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:46:50.899018 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.898968 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 21:46:50.949794 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.936580 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:46:50.950159 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:50.950106 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 21:46:51.089851 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.071253 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80847094-3884-43b5-8145-78f14994e864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a99bb5-qcw99\" (UID: \"80847094-3884-43b5-8145-78f14994e864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" Nov 03 21:46:51.089851 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.071320 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx7wb\" (UniqueName: \"kubernetes.io/projected/80847094-3884-43b5-8145-78f14994e864-kube-api-access-cx7wb\") pod \"maas-api-revoke-sub-13a99bb5-qcw99\" (UID: \"80847094-3884-43b5-8145-78f14994e864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" Nov 03 21:46:51.089851 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.071714 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80847094-3884-43b5-8145-78f14994e864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a99bb5-qcw99\" (UID: \"80847094-3884-43b5-8145-78f14994e864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" Nov 03 21:46:51.111114 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.103838 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:51.236559 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.232894 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80847094-3884-43b5-8145-78f14994e864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a99bb5-qcw99\" (UID: \"80847094-3884-43b5-8145-78f14994e864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" Nov 03 21:46:51.280149 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.267552 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80847094-3884-43b5-8145-78f14994e864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a99bb5-qcw99\" (UID: \"80847094-3884-43b5-8145-78f14994e864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" Nov 03 21:46:51.293129 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.293085 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 21:46:51.337803 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.332152 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:51.337803 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.293018 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:51.360740 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.349237 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:46:51.361054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.360993 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:46:51.362385 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.362340 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db" podUID="6fe7207f-2102-4312-b28d-9c91c22b0ddb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:51.493696 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.477897 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx7wb\" (UniqueName: \"kubernetes.io/projected/80847094-3884-43b5-8145-78f14994e864-kube-api-access-cx7wb\") pod \"maas-api-revoke-sub-13a99bb5-qcw99\" (UID: \"80847094-3884-43b5-8145-78f14994e864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" Nov 03 21:46:51.505714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.500714 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:46:51.505714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.500789 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 21:46:51.505714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.500908 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:46:51.505714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.500933 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 21:46:51.603906 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.603840 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:51.675745 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.633094 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:46:51.712694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.709649 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 21:46:51.712694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.633135 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:46:51.712694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.709981 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 21:46:51.733686 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:46:51.731449 5041 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.098s" Nov 03 21:46:51.733686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.731543 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:46:51.742689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.738763 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:46:51.742689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.738827 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 21:46:51.756779 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.756750 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:46:51.785946 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.785796 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:46:51.785946 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.785862 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 21:46:51.838899 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.838821 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:51.842951 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.842907 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:46:51.879709 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:51.861256 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" Nov 03 21:46:52.194171 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.194073 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:52.293720 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.293653 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:52.592288 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.336106 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:52.596682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.336142 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 21:46:52.627176 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.359874 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:46:52.628012 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.367786 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:46:52.628380 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.424167 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:52.734812 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.505677 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:46:52.851426 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.847762 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 21:46:52.851426 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.675828 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 21:46:52.886727 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.884616 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:46:52.900395 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.626805 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:52.909789 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.510787 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:46:52.924784 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.924736 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 21:46:52.925000 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.708075 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:46:52.950680 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.925115 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 21:46:52.950988 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.835143 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:46:52.951116 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:52.951088 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 21:46:53.195687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:53.195256 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:53.469753 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:53.469714 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:46:53.470036 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:53.470008 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 21:46:53.470200 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:53.470185 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:46:53.470303 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:53.470286 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 21:46:53.593499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:53.593453 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:53.617798 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:53.617745 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:53.674684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:53.650804 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:53.675770 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:53.650852 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:53.676573 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:53.650873 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 21:46:53.736950 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:53.736858 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:46:53.737212 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:53.737179 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 21:46:53.754175 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:53.754140 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:46:53.758765 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:53.758728 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 21:46:53.764015 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:46:53.763986 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454e2120_320f_4213_b2f5_0c0034a5f05d.slice/crio-512211059af4965c9ec03c15cf9ddbf4afbee14ab4d53d6dcb805f4e1b1ebe70 WatchSource:0}: Error finding container 512211059af4965c9ec03c15cf9ddbf4afbee14ab4d53d6dcb805f4e1b1ebe70: Status 404 returned error can't find the container with id 512211059af4965c9ec03c15cf9ddbf4afbee14ab4d53d6dcb805f4e1b1ebe70 Nov 03 21:46:53.797051 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:53.797003 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:53.854052 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:53.854008 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:54.273294 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:54.273248 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:54.456312 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:54.456272 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:46:54.456617 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:54.456586 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 21:46:54.460201 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:54.460154 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:46:54.460408 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:54.460384 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 21:46:54.774499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:54.774451 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 21:46:54.775451 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:54.775417 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:46:54.775898 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:54.775874 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:54.776302 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:54.776274 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:54.838692 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:54.828989 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:54.896320 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:54.896272 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:46:54.896608 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:54.896574 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 21:46:55.296258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:54.929905 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:46:55.296514 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.296488 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 21:46:55.296626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.110467 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 21:46:55.319943 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:54.965049 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 21:46:55.370047 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.288498 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 21:46:55.370303 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.288279 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:55.805891 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.803321 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 21:46:55.805891 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.803425 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:46:55.805891 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.803471 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:46:55.847813 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:46:55.847731 5041 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.046s" Nov 03 21:46:55.848122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.848094 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" event={"ID":"80847094-3884-43b5-8145-78f14994e864","Type":"ContainerStarted","Data":"803ac870fc5e5a95c755a78024e50df35452d87455276210aa792315f5ff6186"} Nov 03 21:46:55.848256 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.848240 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh" event={"ID":"454e2120-320f-4213-b2f5-0c0034a5f05d","Type":"ContainerStarted","Data":"512211059af4965c9ec03c15cf9ddbf4afbee14ab4d53d6dcb805f4e1b1ebe70"} Nov 03 21:46:55.860377 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.860350 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:46:55.860600 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.860583 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:46:55.860734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.860721 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:55.885689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.860844 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:55.886214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.865558 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 21:46:55.886214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.886028 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 21:46:55.886214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.865637 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:46:55.886214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.886094 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 21:46:55.886214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.874875 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:46:55.886214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.886137 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 21:46:55.886214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.874914 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:46:55.886214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.886179 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 21:46:55.907707 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.898746 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:46:55.966473 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.966444 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:55.966760 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.966745 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:55.985945 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:55.985914 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:46:56.052637 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.052610 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:46:56.093036 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.083918 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:46:56.203499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.195167 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:46:56.203499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.203371 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:46:56.204493 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.204475 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:46:56.210252 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.210232 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:46:56.241774 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.241736 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh"] Nov 03 21:46:56.298065 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.298033 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99"] Nov 03 21:46:56.315967 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.315941 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6"] Nov 03 21:46:56.374688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.370277 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6"] Nov 03 21:46:56.374688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.370380 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:56.388189 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.376958 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.479742 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.413725 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-workload-certs\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.479742 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.413800 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d409dea-7691-4be3-9769-5e196850a3f2-istio-token\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.479742 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.413965 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d409dea-7691-4be3-9769-5e196850a3f2-istio-podinfo\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.479742 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.414069 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-workload-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.479742 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.414141 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4f4\" (UniqueName: \"kubernetes.io/projected/7d409dea-7691-4be3-9769-5e196850a3f2-kube-api-access-vn4f4\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.479742 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.414195 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-istio-data\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.479742 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.414296 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d409dea-7691-4be3-9769-5e196850a3f2-istiod-ca-cert\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.479742 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.414339 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-credential-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.479742 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.414364 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-istio-envoy\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.515921 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.515888 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:46:56.516194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.516163 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.529789 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d409dea-7691-4be3-9769-5e196850a3f2-istio-podinfo\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.529909 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-workload-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.529982 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4f4\" (UniqueName: \"kubernetes.io/projected/7d409dea-7691-4be3-9769-5e196850a3f2-kube-api-access-vn4f4\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.530038 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-istio-data\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.530133 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d409dea-7691-4be3-9769-5e196850a3f2-istiod-ca-cert\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.530174 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-credential-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.530201 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-istio-envoy\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.530251 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-workload-certs\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.530304 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d409dea-7691-4be3-9769-5e196850a3f2-istio-token\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.535698 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.535749 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.536755 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.537547 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d409dea-7691-4be3-9769-5e196850a3f2-istiod-ca-cert\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.537849 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-workload-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.538376 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-istio-data\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.543053 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-workload-certs\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.543542 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-credential-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.594906 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-istio-envoy\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.603730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.598137 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d409dea-7691-4be3-9769-5e196850a3f2-istio-token\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.607481 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.607423 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4f4\" (UniqueName: \"kubernetes.io/projected/7d409dea-7691-4be3-9769-5e196850a3f2-kube-api-access-vn4f4\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.616757 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.616370 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d409dea-7691-4be3-9769-5e196850a3f2-istio-podinfo\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:56.665011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.664978 5041 generic.go:356] "Generic (PLEG): container finished" podID="454e2120-320f-4213-b2f5-0c0034a5f05d" containerID="d3c1e7a091849f80152306c3ff905974091b4571276ad13761b0688061087ba6" exitCode=0 Nov 03 21:46:56.700411 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.700377 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 21:46:56.700720 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.700631 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 21:46:56.709539 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.709118 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" event={"ID":"80847094-3884-43b5-8145-78f14994e864","Type":"ContainerStarted","Data":"6cba811328011ece107f3f8f1a989a87bfe8a1c13276029f25f6eb2d3ec88a6d"} Nov 03 21:46:56.711840 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.711811 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh" event={"ID":"454e2120-320f-4213-b2f5-0c0034a5f05d","Type":"ContainerDied","Data":"d3c1e7a091849f80152306c3ff905974091b4571276ad13761b0688061087ba6"} Nov 03 21:46:56.866735 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:56.866614 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:57.606514 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:57.606434 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:57.664014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:57.663984 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:46:57.674210 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:57.674074 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" podStartSLOduration=9.674054945 podStartE2EDuration="9.674054945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:57.232435803 +0000 UTC m=+1032.739664108" watchObservedRunningTime="2026-11-03 21:46:57.674054945 +0000 UTC m=+1033.181283226" Nov 03 21:46:57.778047 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:57.710702 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:57.778047 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:57.720705 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:46:57.778047 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:57.755264 5041 generic.go:356] "Generic (PLEG): container finished" podID="80847094-3884-43b5-8145-78f14994e864" containerID="6cba811328011ece107f3f8f1a989a87bfe8a1c13276029f25f6eb2d3ec88a6d" exitCode=0 Nov 03 21:46:57.778047 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:57.758085 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" event={"ID":"80847094-3884-43b5-8145-78f14994e864","Type":"ContainerDied","Data":"6cba811328011ece107f3f8f1a989a87bfe8a1c13276029f25f6eb2d3ec88a6d"} Nov 03 21:46:57.798520 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:57.796036 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:57.806057 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:57.805970 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:46:58.066562 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.066528 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6"] Nov 03 21:46:58.129140 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.129100 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:58.129403 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.129381 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:58.129520 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.129506 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:46:58.272692 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.272606 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh" Nov 03 21:46:58.356970 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.356932 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/454e2120-320f-4213-b2f5-0c0034a5f05d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/454e2120-320f-4213-b2f5-0c0034a5f05d-maas-api-xks-ca\") pod \"454e2120-320f-4213-b2f5-0c0034a5f05d\" (UID: \"454e2120-320f-4213-b2f5-0c0034a5f05d\") " Nov 03 21:46:58.357247 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.357222 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/454e2120-320f-4213-b2f5-0c0034a5f05d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/454e2120-320f-4213-b2f5-0c0034a5f05d-maas-api-openshift-ca\") pod \"454e2120-320f-4213-b2f5-0c0034a5f05d\" (UID: \"454e2120-320f-4213-b2f5-0c0034a5f05d\") " Nov 03 21:46:58.357358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.357278 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454e2120-320f-4213-b2f5-0c0034a5f05d-kube-api-access-89bpg\" (UniqueName: \"kubernetes.io/projected/454e2120-320f-4213-b2f5-0c0034a5f05d-kube-api-access-89bpg\") pod \"454e2120-320f-4213-b2f5-0c0034a5f05d\" (UID: \"454e2120-320f-4213-b2f5-0c0034a5f05d\") " Nov 03 21:46:58.361187 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.361131 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454e2120-320f-4213-b2f5-0c0034a5f05d-kube-api-access-89bpg" pod "454e2120-320f-4213-b2f5-0c0034a5f05d" (UID: "454e2120-320f-4213-b2f5-0c0034a5f05d"). InnerVolumeSpecName "kube-api-access-89bpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:46:58.361551 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.361524 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454e2120-320f-4213-b2f5-0c0034a5f05d-maas-api-openshift-ca" pod "454e2120-320f-4213-b2f5-0c0034a5f05d" (UID: "454e2120-320f-4213-b2f5-0c0034a5f05d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:46:58.362880 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.362849 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/454e2120-320f-4213-b2f5-0c0034a5f05d-maas-api-xks-ca" pod "454e2120-320f-4213-b2f5-0c0034a5f05d" (UID: "454e2120-320f-4213-b2f5-0c0034a5f05d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:46:58.373989 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.372446 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:58.376501 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.376477 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:46:58.459688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.459602 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89bpg\" (UniqueName: \"kubernetes.io/projected/454e2120-320f-4213-b2f5-0c0034a5f05d-kube-api-access-89bpg\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:58.459688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.459636 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/454e2120-320f-4213-b2f5-0c0034a5f05d-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:58.459688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.459652 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/454e2120-320f-4213-b2f5-0c0034a5f05d-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:58.769973 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.769938 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" event={"ID":"7d409dea-7691-4be3-9769-5e196850a3f2","Type":"ContainerStarted","Data":"46e44e5d19240a478c4a2a4b0e82b2d3b0f7f093805adfc431f8bf89705aa7d7"} Nov 03 21:46:58.770139 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.769998 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" event={"ID":"7d409dea-7691-4be3-9769-5e196850a3f2","Type":"ContainerStarted","Data":"4107fbc8533ca0ad0841c241866bdad83a4b69a37df8157ba6a90bfe51866034"} Nov 03 21:46:58.771398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.771375 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh" Nov 03 21:46:58.771398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.771376 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh" event={"ID":"454e2120-320f-4213-b2f5-0c0034a5f05d","Type":"ContainerDied","Data":"512211059af4965c9ec03c15cf9ddbf4afbee14ab4d53d6dcb805f4e1b1ebe70"} Nov 03 21:46:58.771588 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.771428 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="512211059af4965c9ec03c15cf9ddbf4afbee14ab4d53d6dcb805f4e1b1ebe70" Nov 03 21:46:58.869305 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.868184 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:46:58.992918 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:58.992893 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" Nov 03 21:46:59.064961 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:59.064904 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" podStartSLOduration=4.064887942 podStartE2EDuration="4.064887942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:58.822250015 +0000 UTC m=+1034.329478321" watchObservedRunningTime="2026-11-03 21:46:59.064887942 +0000 UTC m=+1034.572116222" Nov 03 21:46:59.069643 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:59.069613 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80847094-3884-43b5-8145-78f14994e864-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80847094-3884-43b5-8145-78f14994e864-maas-api-openshift-ca\") pod \"80847094-3884-43b5-8145-78f14994e864\" (UID: \"80847094-3884-43b5-8145-78f14994e864\") " Nov 03 21:46:59.070014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:59.069746 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80847094-3884-43b5-8145-78f14994e864-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80847094-3884-43b5-8145-78f14994e864-maas-api-xks-ca\") pod \"80847094-3884-43b5-8145-78f14994e864\" (UID: \"80847094-3884-43b5-8145-78f14994e864\") " Nov 03 21:46:59.070014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:59.069889 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80847094-3884-43b5-8145-78f14994e864-kube-api-access-cx7wb\" (UniqueName: \"kubernetes.io/projected/80847094-3884-43b5-8145-78f14994e864-kube-api-access-cx7wb\") pod \"80847094-3884-43b5-8145-78f14994e864\" (UID: \"80847094-3884-43b5-8145-78f14994e864\") " Nov 03 21:46:59.070130 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:59.070018 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80847094-3884-43b5-8145-78f14994e864-maas-api-openshift-ca" pod "80847094-3884-43b5-8145-78f14994e864" (UID: "80847094-3884-43b5-8145-78f14994e864"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:46:59.070624 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:59.070599 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80847094-3884-43b5-8145-78f14994e864-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:59.072118 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:59.072083 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80847094-3884-43b5-8145-78f14994e864-kube-api-access-cx7wb" pod "80847094-3884-43b5-8145-78f14994e864" (UID: "80847094-3884-43b5-8145-78f14994e864"). InnerVolumeSpecName "kube-api-access-cx7wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:46:59.072495 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:59.072472 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80847094-3884-43b5-8145-78f14994e864-maas-api-xks-ca" pod "80847094-3884-43b5-8145-78f14994e864" (UID: "80847094-3884-43b5-8145-78f14994e864"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:46:59.175024 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:59.174983 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80847094-3884-43b5-8145-78f14994e864-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:59.175280 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:59.175257 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx7wb\" (UniqueName: \"kubernetes.io/projected/80847094-3884-43b5-8145-78f14994e864-kube-api-access-cx7wb\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:46:59.783014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:59.782982 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" Nov 03 21:46:59.783014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:59.782990 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99" event={"ID":"80847094-3884-43b5-8145-78f14994e864","Type":"ContainerDied","Data":"803ac870fc5e5a95c755a78024e50df35452d87455276210aa792315f5ff6186"} Nov 03 21:46:59.783226 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:59.783049 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="803ac870fc5e5a95c755a78024e50df35452d87455276210aa792315f5ff6186" Nov 03 21:46:59.869867 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:59.869832 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 21:46:59.870014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:46:59.869881 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" podUID="7d409dea-7691-4be3-9769-5e196850a3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 21:47:00.869010 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:00.868967 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 21:47:00.869421 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:00.869032 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" podUID="7d409dea-7691-4be3-9769-5e196850a3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 21:47:01.244318 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.244281 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4"] Nov 03 21:47:01.245723 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.245694 5041 cpu_manager.go:414] "containerMap: removing container" podUID="80847094-3884-43b5-8145-78f14994e864" containerName="revoke-keys" Nov 03 21:47:01.245723 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.245720 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80847094-3884-43b5-8145-78f14994e864" containerName="revoke-keys" Nov 03 21:47:01.245913 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.245737 5041 cpu_manager.go:414] "containerMap: removing container" podUID="454e2120-320f-4213-b2f5-0c0034a5f05d" containerName="revoke-keys" Nov 03 21:47:01.245913 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.245746 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="454e2120-320f-4213-b2f5-0c0034a5f05d" containerName="revoke-keys" Nov 03 21:47:01.246046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.246026 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="80847094-3884-43b5-8145-78f14994e864" containerName="revoke-keys" Nov 03 21:47:01.246214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.246195 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="454e2120-320f-4213-b2f5-0c0034a5f05d" containerName="revoke-keys" Nov 03 21:47:01.270887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.270850 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4"] Nov 03 21:47:01.271144 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.271117 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" Nov 03 21:47:01.287686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.287633 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:47:01.288450 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.288336 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:47:01.477687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.469851 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81b36a7b-q86r4\" (UID: \"a26f5bfa-3760-40eb-b7cf-d5da5e18271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" Nov 03 21:47:01.477687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.469915 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81b36a7b-q86r4\" (UID: \"a26f5bfa-3760-40eb-b7cf-d5da5e18271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" Nov 03 21:47:01.477687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.470215 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qkfq\" (UniqueName: \"kubernetes.io/projected/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-kube-api-access-5qkfq\") pod \"maas-api-revoke-sub-81b36a7b-q86r4\" (UID: \"a26f5bfa-3760-40eb-b7cf-d5da5e18271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" Nov 03 21:47:01.578740 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.571503 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81b36a7b-q86r4\" (UID: \"a26f5bfa-3760-40eb-b7cf-d5da5e18271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" Nov 03 21:47:01.578740 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.571559 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81b36a7b-q86r4\" (UID: \"a26f5bfa-3760-40eb-b7cf-d5da5e18271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" Nov 03 21:47:01.578740 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.571816 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qkfq\" (UniqueName: \"kubernetes.io/projected/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-kube-api-access-5qkfq\") pod \"maas-api-revoke-sub-81b36a7b-q86r4\" (UID: \"a26f5bfa-3760-40eb-b7cf-d5da5e18271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" Nov 03 21:47:01.578740 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.573356 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81b36a7b-q86r4\" (UID: \"a26f5bfa-3760-40eb-b7cf-d5da5e18271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" Nov 03 21:47:01.611683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.609176 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81b36a7b-q86r4\" (UID: \"a26f5bfa-3760-40eb-b7cf-d5da5e18271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" Nov 03 21:47:01.611683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.609391 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qkfq\" (UniqueName: \"kubernetes.io/projected/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-kube-api-access-5qkfq\") pod \"maas-api-revoke-sub-81b36a7b-q86r4\" (UID: \"a26f5bfa-3760-40eb-b7cf-d5da5e18271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" Nov 03 21:47:01.877734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.877101 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 21:47:01.877734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.877171 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" podUID="7d409dea-7691-4be3-9769-5e196850a3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 21:47:01.901687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:01.897932 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" Nov 03 21:47:02.393684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:02.382882 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:02.627687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:02.626799 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:02.635687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:02.632072 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:02.871140 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:02.862775 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:02.931821 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:02.879407 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 21:47:02.931821 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:02.879485 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" podUID="7d409dea-7691-4be3-9769-5e196850a3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 21:47:02.931821 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:02.910212 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 03 21:47:03.108036 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:03.107108 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 21:47:03.142731 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:03.141255 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:03.431749 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:03.431020 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:03.494454 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:03.432230 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 21:47:03.494454 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:03.432458 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 21:47:03.494454 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:03.433046 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:47:03.645726 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:03.636795 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:03.719732 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:03.709242 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:47:03.895691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:03.869262 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 21:47:03.895691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:03.870643 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:47:03.898149 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:03.896769 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 21:47:03.898149 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:03.896830 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" podUID="7d409dea-7691-4be3-9769-5e196850a3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 21:47:03.979235 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:03.943375 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:03.979235 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:03.976162 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:47:04.022736 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:47:04.016893 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26f5bfa_3760_40eb_b7cf_d5da5e18271a.slice/crio-790349a1cc045ffd71211f578590e23d42787eedf5eb70eb7cb3038c948d097b WatchSource:0}: Error finding container 790349a1cc045ffd71211f578590e23d42787eedf5eb70eb7cb3038c948d097b: Status 404 returned error can't find the container with id 790349a1cc045ffd71211f578590e23d42787eedf5eb70eb7cb3038c948d097b Nov 03 21:47:04.127910 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:04.114043 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:04.128332 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:04.128286 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:47:04.218688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:04.217787 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:04.218688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:04.218684 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:47:04.469676 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:04.440395 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.120:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:04.469676 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:04.442752 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:04.469676 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:04.444470 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:04.469676 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:04.444795 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:04.469676 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:04.445095 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:47:04.469676 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:04.445153 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:47:04.709028 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:04.645779 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:04.896788 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:04.870811 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 21:47:04.975746 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:04.975622 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 03 21:47:05.071394 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:05.014815 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 21:47:05.087583 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:05.087531 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" podUID="7d409dea-7691-4be3-9769-5e196850a3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 21:47:05.267944 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:05.216878 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:05.267944 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:05.252784 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:05.349801 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:05.270036 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 03 21:47:05.560626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:05.520810 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.120:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:05.725689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:05.717648 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:05.752688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:05.728024 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 21:47:05.753093 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:05.718541 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:05.882789 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:05.823505 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:05.910191 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:05.887051 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 21:47:05.910191 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:05.902816 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" podUID="7d409dea-7691-4be3-9769-5e196850a3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 21:47:05.921684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:05.920868 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:06.054692 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.049234 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:06.205859 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:47:06.203199 5041 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.153s" Nov 03 21:47:06.205859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.203330 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" event={"ID":"a26f5bfa-3760-40eb-b7cf-d5da5e18271a","Type":"ContainerStarted","Data":"790349a1cc045ffd71211f578590e23d42787eedf5eb70eb7cb3038c948d097b"} Nov 03 21:47:06.205859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.203446 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:47:06.222954 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.222903 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:06.256746 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.256685 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:06.336969 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.333864 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4"] Nov 03 21:47:06.411303 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.411269 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:47:06.423482 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.423446 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:47:06.494090 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.493066 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:47:06.546390 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.545784 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:47:06.546390 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.545869 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:47:06.547931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.547909 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:47:06.728444 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.728397 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:06.794874 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.794778 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:06.828451 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.808292 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:47:06.890640 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.886084 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 21:47:06.890640 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.886159 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" podUID="7d409dea-7691-4be3-9769-5e196850a3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 21:47:06.905341 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.903015 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:47:06.905341 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.903091 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:47:06.907823 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:06.907794 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" event={"ID":"a26f5bfa-3760-40eb-b7cf-d5da5e18271a","Type":"ContainerStarted","Data":"81c57b9719cfdc2e4abd53351a9f876070cff142eb5913fdf35474adccb72efd"} Nov 03 21:47:07.092410 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:07.092292 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" podStartSLOduration=6.09227843 podStartE2EDuration="6.09227843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:07.091393129 +0000 UTC m=+1042.598621433" watchObservedRunningTime="2026-11-03 21:47:07.09227843 +0000 UTC m=+1042.599506710" Nov 03 21:47:07.869390 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:07.869351 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 21:47:07.869856 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:07.869421 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" podUID="7d409dea-7691-4be3-9769-5e196850a3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 21:47:07.920028 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:07.919986 5041 generic.go:356] "Generic (PLEG): container finished" podID="a26f5bfa-3760-40eb-b7cf-d5da5e18271a" containerID="81c57b9719cfdc2e4abd53351a9f876070cff142eb5913fdf35474adccb72efd" exitCode=0 Nov 03 21:47:07.920235 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:07.920064 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" event={"ID":"a26f5bfa-3760-40eb-b7cf-d5da5e18271a","Type":"ContainerDied","Data":"81c57b9719cfdc2e4abd53351a9f876070cff142eb5913fdf35474adccb72efd"} Nov 03 21:47:08.310024 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:08.309988 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd"] Nov 03 21:47:08.315423 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:08.315399 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" Nov 03 21:47:08.332452 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:08.332417 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd"] Nov 03 21:47:08.414187 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:08.414144 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcmr6\" (UniqueName: \"kubernetes.io/projected/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-kube-api-access-xcmr6\") pod \"maas-api-revoke-sub-7fbcb306-qqffd\" (UID: \"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" Nov 03 21:47:08.414380 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:08.414322 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fbcb306-qqffd\" (UID: \"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" Nov 03 21:47:08.414571 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:08.414553 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fbcb306-qqffd\" (UID: \"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" Nov 03 21:47:08.516340 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:08.516296 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fbcb306-qqffd\" (UID: \"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" Nov 03 21:47:08.516558 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:08.516486 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcmr6\" (UniqueName: \"kubernetes.io/projected/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-kube-api-access-xcmr6\") pod \"maas-api-revoke-sub-7fbcb306-qqffd\" (UID: \"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" Nov 03 21:47:08.516641 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:08.516613 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fbcb306-qqffd\" (UID: \"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" Nov 03 21:47:08.517309 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:08.517279 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fbcb306-qqffd\" (UID: \"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" Nov 03 21:47:08.518929 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:08.518899 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fbcb306-qqffd\" (UID: \"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" Nov 03 21:47:08.538152 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:08.538116 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcmr6\" (UniqueName: \"kubernetes.io/projected/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-kube-api-access-xcmr6\") pod \"maas-api-revoke-sub-7fbcb306-qqffd\" (UID: \"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" Nov 03 21:47:08.652375 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:08.652276 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" Nov 03 21:47:08.947236 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:08.947072 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:47:08.947906 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:08.947881 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:47:08.949158 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:08.949134 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:47:09.005000 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:47:09.004949 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9cd6fb2_502f_4504_8d0d_b880dbcb20b9.slice/crio-67bfea64977d30b7e9c762b13604a4a5c31f06fcf36e510b29a371db460fcf62 WatchSource:0}: Error finding container 67bfea64977d30b7e9c762b13604a4a5c31f06fcf36e510b29a371db460fcf62: Status 404 returned error can't find the container with id 67bfea64977d30b7e9c762b13604a4a5c31f06fcf36e510b29a371db460fcf62 Nov 03 21:47:09.020602 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.020302 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd"] Nov 03 21:47:09.253652 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.253124 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" Nov 03 21:47:09.346256 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.346146 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-kube-api-access-5qkfq\" (UniqueName: \"kubernetes.io/projected/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-kube-api-access-5qkfq\") pod \"a26f5bfa-3760-40eb-b7cf-d5da5e18271a\" (UID: \"a26f5bfa-3760-40eb-b7cf-d5da5e18271a\") " Nov 03 21:47:09.346256 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.346260 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-maas-api-xks-ca\") pod \"a26f5bfa-3760-40eb-b7cf-d5da5e18271a\" (UID: \"a26f5bfa-3760-40eb-b7cf-d5da5e18271a\") " Nov 03 21:47:09.346642 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.346605 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-maas-api-openshift-ca\") pod \"a26f5bfa-3760-40eb-b7cf-d5da5e18271a\" (UID: \"a26f5bfa-3760-40eb-b7cf-d5da5e18271a\") " Nov 03 21:47:09.347545 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.347207 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-maas-api-openshift-ca" pod "a26f5bfa-3760-40eb-b7cf-d5da5e18271a" (UID: "a26f5bfa-3760-40eb-b7cf-d5da5e18271a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:09.347545 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.347514 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:09.408560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.408516 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-maas-api-xks-ca" pod "a26f5bfa-3760-40eb-b7cf-d5da5e18271a" (UID: "a26f5bfa-3760-40eb-b7cf-d5da5e18271a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:09.408773 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.408605 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-kube-api-access-5qkfq" pod "a26f5bfa-3760-40eb-b7cf-d5da5e18271a" (UID: "a26f5bfa-3760-40eb-b7cf-d5da5e18271a"). InnerVolumeSpecName "kube-api-access-5qkfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:09.449646 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.449608 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qkfq\" (UniqueName: \"kubernetes.io/projected/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-kube-api-access-5qkfq\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:09.449646 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.449646 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a26f5bfa-3760-40eb-b7cf-d5da5e18271a-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:09.941852 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.941814 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" event={"ID":"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9","Type":"ContainerStarted","Data":"680b5d6f97f0d63266206268f29eb6927c6bea1155b4d6d843b0ad80f7a1fc1e"} Nov 03 21:47:09.942005 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.941870 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" event={"ID":"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9","Type":"ContainerStarted","Data":"67bfea64977d30b7e9c762b13604a4a5c31f06fcf36e510b29a371db460fcf62"} Nov 03 21:47:09.943809 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.943774 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" Nov 03 21:47:09.943956 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.943834 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4" event={"ID":"a26f5bfa-3760-40eb-b7cf-d5da5e18271a","Type":"ContainerDied","Data":"790349a1cc045ffd71211f578590e23d42787eedf5eb70eb7cb3038c948d097b"} Nov 03 21:47:09.943956 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.943893 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="790349a1cc045ffd71211f578590e23d42787eedf5eb70eb7cb3038c948d097b" Nov 03 21:47:09.982311 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:09.982249 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" podStartSLOduration=1.982229493 podStartE2EDuration="1.982229493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:09.982103979 +0000 UTC m=+1045.489332260" watchObservedRunningTime="2026-11-03 21:47:09.982229493 +0000 UTC m=+1045.489457774" Nov 03 21:47:10.780281 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.780245 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm"] Nov 03 21:47:10.781104 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.781082 5041 cpu_manager.go:414] "containerMap: removing container" podUID="a26f5bfa-3760-40eb-b7cf-d5da5e18271a" containerName="revoke-keys" Nov 03 21:47:10.781104 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.781102 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a26f5bfa-3760-40eb-b7cf-d5da5e18271a" containerName="revoke-keys" Nov 03 21:47:10.781270 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.781217 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a26f5bfa-3760-40eb-b7cf-d5da5e18271a" containerName="revoke-keys" Nov 03 21:47:10.781819 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:47:10.781791 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:47:10.785278 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.785260 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" Nov 03 21:47:10.801684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.801100 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm"] Nov 03 21:47:10.874093 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.874052 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkf2s\" (UniqueName: \"kubernetes.io/projected/d0be0f94-8035-4087-be7b-1a988edecae5-kube-api-access-zkf2s\") pod \"maas-api-revoke-sub-68d768a4-q7pwm\" (UID: \"d0be0f94-8035-4087-be7b-1a988edecae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" Nov 03 21:47:10.874283 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.874126 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0be0f94-8035-4087-be7b-1a988edecae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68d768a4-q7pwm\" (UID: \"d0be0f94-8035-4087-be7b-1a988edecae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" Nov 03 21:47:10.874283 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.874211 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0be0f94-8035-4087-be7b-1a988edecae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68d768a4-q7pwm\" (UID: \"d0be0f94-8035-4087-be7b-1a988edecae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" Nov 03 21:47:10.955134 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.955097 5041 generic.go:356] "Generic (PLEG): container finished" podID="d9cd6fb2-502f-4504-8d0d-b880dbcb20b9" containerID="680b5d6f97f0d63266206268f29eb6927c6bea1155b4d6d843b0ad80f7a1fc1e" exitCode=0 Nov 03 21:47:10.955331 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.955162 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" event={"ID":"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9","Type":"ContainerDied","Data":"680b5d6f97f0d63266206268f29eb6927c6bea1155b4d6d843b0ad80f7a1fc1e"} Nov 03 21:47:10.975709 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.975632 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkf2s\" (UniqueName: \"kubernetes.io/projected/d0be0f94-8035-4087-be7b-1a988edecae5-kube-api-access-zkf2s\") pod \"maas-api-revoke-sub-68d768a4-q7pwm\" (UID: \"d0be0f94-8035-4087-be7b-1a988edecae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" Nov 03 21:47:10.975939 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.975731 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0be0f94-8035-4087-be7b-1a988edecae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68d768a4-q7pwm\" (UID: \"d0be0f94-8035-4087-be7b-1a988edecae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" Nov 03 21:47:10.975939 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.975764 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0be0f94-8035-4087-be7b-1a988edecae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68d768a4-q7pwm\" (UID: \"d0be0f94-8035-4087-be7b-1a988edecae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" Nov 03 21:47:10.976748 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.976718 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0be0f94-8035-4087-be7b-1a988edecae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68d768a4-q7pwm\" (UID: \"d0be0f94-8035-4087-be7b-1a988edecae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" Nov 03 21:47:10.978517 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.978492 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0be0f94-8035-4087-be7b-1a988edecae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68d768a4-q7pwm\" (UID: \"d0be0f94-8035-4087-be7b-1a988edecae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" Nov 03 21:47:10.999764 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:10.999729 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkf2s\" (UniqueName: \"kubernetes.io/projected/d0be0f94-8035-4087-be7b-1a988edecae5-kube-api-access-zkf2s\") pod \"maas-api-revoke-sub-68d768a4-q7pwm\" (UID: \"d0be0f94-8035-4087-be7b-1a988edecae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" Nov 03 21:47:11.108283 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:11.108192 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" Nov 03 21:47:11.428130 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:47:11.428099 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0be0f94_8035_4087_be7b_1a988edecae5.slice/crio-cb1b898d43bf9da26d81c5e8ffa7389c59dfa6c6c4e0484f46e413c4518cb288 WatchSource:0}: Error finding container cb1b898d43bf9da26d81c5e8ffa7389c59dfa6c6c4e0484f46e413c4518cb288: Status 404 returned error can't find the container with id cb1b898d43bf9da26d81c5e8ffa7389c59dfa6c6c4e0484f46e413c4518cb288 Nov 03 21:47:11.428979 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:11.428952 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm"] Nov 03 21:47:11.972259 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:11.972167 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" event={"ID":"d0be0f94-8035-4087-be7b-1a988edecae5","Type":"ContainerStarted","Data":"7de9237a0a0e27a0945e40d6e059832d9b354de855dbbc621a22c1794c110399"} Nov 03 21:47:11.972259 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:11.972233 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" event={"ID":"d0be0f94-8035-4087-be7b-1a988edecae5","Type":"ContainerStarted","Data":"cb1b898d43bf9da26d81c5e8ffa7389c59dfa6c6c4e0484f46e413c4518cb288"} Nov 03 21:47:12.016366 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:12.016309 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" podStartSLOduration=2.016293724 podStartE2EDuration="2.016293724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:12.011955154 +0000 UTC m=+1047.519183435" watchObservedRunningTime="2026-11-03 21:47:12.016293724 +0000 UTC m=+1047.523522004" Nov 03 21:47:12.162629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:12.162604 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" Nov 03 21:47:12.301055 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:12.300956 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-kube-api-access-xcmr6\" (UniqueName: \"kubernetes.io/projected/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-kube-api-access-xcmr6\") pod \"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9\" (UID: \"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9\") " Nov 03 21:47:12.301238 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:12.301164 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-maas-api-xks-ca\") pod \"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9\" (UID: \"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9\") " Nov 03 21:47:12.301238 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:12.301194 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-maas-api-openshift-ca\") pod \"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9\" (UID: \"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9\") " Nov 03 21:47:12.301607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:12.301577 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-maas-api-openshift-ca" pod "d9cd6fb2-502f-4504-8d0d-b880dbcb20b9" (UID: "d9cd6fb2-502f-4504-8d0d-b880dbcb20b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:12.301929 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:12.301903 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:12.303262 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:12.303239 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-kube-api-access-xcmr6" pod "d9cd6fb2-502f-4504-8d0d-b880dbcb20b9" (UID: "d9cd6fb2-502f-4504-8d0d-b880dbcb20b9"). InnerVolumeSpecName "kube-api-access-xcmr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:12.303426 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:12.303381 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-maas-api-xks-ca" pod "d9cd6fb2-502f-4504-8d0d-b880dbcb20b9" (UID: "d9cd6fb2-502f-4504-8d0d-b880dbcb20b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:12.404038 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:12.404001 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:12.404038 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:12.404035 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcmr6\" (UniqueName: \"kubernetes.io/projected/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9-kube-api-access-xcmr6\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:12.985387 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:12.985353 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" Nov 03 21:47:12.987214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:12.986101 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd" event={"ID":"d9cd6fb2-502f-4504-8d0d-b880dbcb20b9","Type":"ContainerDied","Data":"67bfea64977d30b7e9c762b13604a4a5c31f06fcf36e510b29a371db460fcf62"} Nov 03 21:47:12.987214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:12.986157 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67bfea64977d30b7e9c762b13604a4a5c31f06fcf36e510b29a371db460fcf62" Nov 03 21:47:12.989085 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:12.989053 5041 generic.go:356] "Generic (PLEG): container finished" podID="d0be0f94-8035-4087-be7b-1a988edecae5" containerID="7de9237a0a0e27a0945e40d6e059832d9b354de855dbbc621a22c1794c110399" exitCode=0 Nov 03 21:47:12.989220 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:12.989128 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" event={"ID":"d0be0f94-8035-4087-be7b-1a988edecae5","Type":"ContainerDied","Data":"7de9237a0a0e27a0945e40d6e059832d9b354de855dbbc621a22c1794c110399"} Nov 03 21:47:14.209379 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:14.209346 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" Nov 03 21:47:14.336201 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:14.336117 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0be0f94-8035-4087-be7b-1a988edecae5-kube-api-access-zkf2s\" (UniqueName: \"kubernetes.io/projected/d0be0f94-8035-4087-be7b-1a988edecae5-kube-api-access-zkf2s\") pod \"d0be0f94-8035-4087-be7b-1a988edecae5\" (UID: \"d0be0f94-8035-4087-be7b-1a988edecae5\") " Nov 03 21:47:14.336369 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:14.336214 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0be0f94-8035-4087-be7b-1a988edecae5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0be0f94-8035-4087-be7b-1a988edecae5-maas-api-xks-ca\") pod \"d0be0f94-8035-4087-be7b-1a988edecae5\" (UID: \"d0be0f94-8035-4087-be7b-1a988edecae5\") " Nov 03 21:47:14.336369 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:14.336290 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0be0f94-8035-4087-be7b-1a988edecae5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0be0f94-8035-4087-be7b-1a988edecae5-maas-api-openshift-ca\") pod \"d0be0f94-8035-4087-be7b-1a988edecae5\" (UID: \"d0be0f94-8035-4087-be7b-1a988edecae5\") " Nov 03 21:47:14.336718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:14.336647 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0be0f94-8035-4087-be7b-1a988edecae5-maas-api-openshift-ca" pod "d0be0f94-8035-4087-be7b-1a988edecae5" (UID: "d0be0f94-8035-4087-be7b-1a988edecae5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:14.336934 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:14.336913 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0be0f94-8035-4087-be7b-1a988edecae5-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:14.338341 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:14.338318 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0be0f94-8035-4087-be7b-1a988edecae5-kube-api-access-zkf2s" pod "d0be0f94-8035-4087-be7b-1a988edecae5" (UID: "d0be0f94-8035-4087-be7b-1a988edecae5"). InnerVolumeSpecName "kube-api-access-zkf2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:14.338443 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:14.338351 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0be0f94-8035-4087-be7b-1a988edecae5-maas-api-xks-ca" pod "d0be0f94-8035-4087-be7b-1a988edecae5" (UID: "d0be0f94-8035-4087-be7b-1a988edecae5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:14.438583 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:14.438550 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkf2s\" (UniqueName: \"kubernetes.io/projected/d0be0f94-8035-4087-be7b-1a988edecae5-kube-api-access-zkf2s\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:14.438583 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:14.438580 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0be0f94-8035-4087-be7b-1a988edecae5-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:15.022084 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:15.022052 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" Nov 03 21:47:15.022264 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:15.022074 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm" event={"ID":"d0be0f94-8035-4087-be7b-1a988edecae5","Type":"ContainerDied","Data":"cb1b898d43bf9da26d81c5e8ffa7389c59dfa6c6c4e0484f46e413c4518cb288"} Nov 03 21:47:15.022264 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:15.022159 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb1b898d43bf9da26d81c5e8ffa7389c59dfa6c6c4e0484f46e413c4518cb288" Nov 03 21:47:15.973740 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:15.973704 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484"] Nov 03 21:47:15.974438 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:15.974423 5041 cpu_manager.go:414] "containerMap: removing container" podUID="d0be0f94-8035-4087-be7b-1a988edecae5" containerName="revoke-keys" Nov 03 21:47:15.974478 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:15.974439 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0be0f94-8035-4087-be7b-1a988edecae5" containerName="revoke-keys" Nov 03 21:47:15.974478 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:15.974455 5041 cpu_manager.go:414] "containerMap: removing container" podUID="d9cd6fb2-502f-4504-8d0d-b880dbcb20b9" containerName="revoke-keys" Nov 03 21:47:15.974478 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:15.974461 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9cd6fb2-502f-4504-8d0d-b880dbcb20b9" containerName="revoke-keys" Nov 03 21:47:15.974594 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:15.974584 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0be0f94-8035-4087-be7b-1a988edecae5" containerName="revoke-keys" Nov 03 21:47:15.974629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:15.974597 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9cd6fb2-502f-4504-8d0d-b880dbcb20b9" containerName="revoke-keys" Nov 03 21:47:15.978615 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:15.978586 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" Nov 03 21:47:15.998956 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:15.998925 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484"] Nov 03 21:47:16.003465 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:16.003436 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:47:16.003618 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:16.003544 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:47:16.166900 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:16.166863 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e56751c-ebda-4b44-8967-10b0e7ecbc81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5108f5c-4g484\" (UID: \"1e56751c-ebda-4b44-8967-10b0e7ecbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" Nov 03 21:47:16.167086 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:16.167065 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e56751c-ebda-4b44-8967-10b0e7ecbc81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5108f5c-4g484\" (UID: \"1e56751c-ebda-4b44-8967-10b0e7ecbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" Nov 03 21:47:16.167287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:16.167263 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxntg\" (UniqueName: \"kubernetes.io/projected/1e56751c-ebda-4b44-8967-10b0e7ecbc81-kube-api-access-lxntg\") pod \"maas-api-revoke-sub-b5108f5c-4g484\" (UID: \"1e56751c-ebda-4b44-8967-10b0e7ecbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" Nov 03 21:47:16.269027 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:16.268944 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e56751c-ebda-4b44-8967-10b0e7ecbc81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5108f5c-4g484\" (UID: \"1e56751c-ebda-4b44-8967-10b0e7ecbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" Nov 03 21:47:16.269176 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:16.269041 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxntg\" (UniqueName: \"kubernetes.io/projected/1e56751c-ebda-4b44-8967-10b0e7ecbc81-kube-api-access-lxntg\") pod \"maas-api-revoke-sub-b5108f5c-4g484\" (UID: \"1e56751c-ebda-4b44-8967-10b0e7ecbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" Nov 03 21:47:16.269176 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:16.269148 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e56751c-ebda-4b44-8967-10b0e7ecbc81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5108f5c-4g484\" (UID: \"1e56751c-ebda-4b44-8967-10b0e7ecbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" Nov 03 21:47:16.269857 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:16.269828 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e56751c-ebda-4b44-8967-10b0e7ecbc81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5108f5c-4g484\" (UID: \"1e56751c-ebda-4b44-8967-10b0e7ecbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" Nov 03 21:47:16.271558 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:16.271539 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e56751c-ebda-4b44-8967-10b0e7ecbc81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5108f5c-4g484\" (UID: \"1e56751c-ebda-4b44-8967-10b0e7ecbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" Nov 03 21:47:16.289815 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:16.289788 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxntg\" (UniqueName: \"kubernetes.io/projected/1e56751c-ebda-4b44-8967-10b0e7ecbc81-kube-api-access-lxntg\") pod \"maas-api-revoke-sub-b5108f5c-4g484\" (UID: \"1e56751c-ebda-4b44-8967-10b0e7ecbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" Nov 03 21:47:16.296266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:16.296238 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" Nov 03 21:47:16.620226 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:47:16.620196 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e56751c_ebda_4b44_8967_10b0e7ecbc81.slice/crio-b89359e6dd2ffbe16d5a06cf06bf1ea85f0a9f2b301664f09c0e45147e662ad8 WatchSource:0}: Error finding container b89359e6dd2ffbe16d5a06cf06bf1ea85f0a9f2b301664f09c0e45147e662ad8: Status 404 returned error can't find the container with id b89359e6dd2ffbe16d5a06cf06bf1ea85f0a9f2b301664f09c0e45147e662ad8 Nov 03 21:47:16.628974 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:16.628943 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484"] Nov 03 21:47:17.050254 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:17.050208 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" event={"ID":"1e56751c-ebda-4b44-8967-10b0e7ecbc81","Type":"ContainerStarted","Data":"3ce6087af1c70a9297e913f3b5b89c4b4af39ff751e74d6da144f99ec3977458"} Nov 03 21:47:17.050779 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:17.050285 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" event={"ID":"1e56751c-ebda-4b44-8967-10b0e7ecbc81","Type":"ContainerStarted","Data":"b89359e6dd2ffbe16d5a06cf06bf1ea85f0a9f2b301664f09c0e45147e662ad8"} Nov 03 21:47:17.093205 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:17.093153 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" podStartSLOduration=2.093138793 podStartE2EDuration="2.093138793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:17.090328447 +0000 UTC m=+1052.597556739" watchObservedRunningTime="2026-11-03 21:47:17.093138793 +0000 UTC m=+1052.600367073" Nov 03 21:47:17.561811 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:17.561778 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx"] Nov 03 21:47:17.569394 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:17.569191 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" Nov 03 21:47:17.587106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:17.586411 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx"] Nov 03 21:47:17.588310 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:17.588273 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9266c8c-0d14-41f6-93ac-aecddca1ff89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx\" (UID: \"d9266c8c-0d14-41f6-93ac-aecddca1ff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" Nov 03 21:47:17.588470 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:17.588379 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khj6x\" (UniqueName: \"kubernetes.io/projected/d9266c8c-0d14-41f6-93ac-aecddca1ff89-kube-api-access-khj6x\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx\" (UID: \"d9266c8c-0d14-41f6-93ac-aecddca1ff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" Nov 03 21:47:17.588612 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:17.588589 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9266c8c-0d14-41f6-93ac-aecddca1ff89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx\" (UID: \"d9266c8c-0d14-41f6-93ac-aecddca1ff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" Nov 03 21:47:17.689645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:17.689610 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khj6x\" (UniqueName: \"kubernetes.io/projected/d9266c8c-0d14-41f6-93ac-aecddca1ff89-kube-api-access-khj6x\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx\" (UID: \"d9266c8c-0d14-41f6-93ac-aecddca1ff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" Nov 03 21:47:17.689800 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:17.689785 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9266c8c-0d14-41f6-93ac-aecddca1ff89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx\" (UID: \"d9266c8c-0d14-41f6-93ac-aecddca1ff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" Nov 03 21:47:17.689947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:17.689927 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9266c8c-0d14-41f6-93ac-aecddca1ff89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx\" (UID: \"d9266c8c-0d14-41f6-93ac-aecddca1ff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" Nov 03 21:47:17.690620 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:17.690599 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9266c8c-0d14-41f6-93ac-aecddca1ff89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx\" (UID: \"d9266c8c-0d14-41f6-93ac-aecddca1ff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" Nov 03 21:47:17.692320 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:17.692299 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9266c8c-0d14-41f6-93ac-aecddca1ff89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx\" (UID: \"d9266c8c-0d14-41f6-93ac-aecddca1ff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" Nov 03 21:47:17.710238 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:17.710202 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khj6x\" (UniqueName: \"kubernetes.io/projected/d9266c8c-0d14-41f6-93ac-aecddca1ff89-kube-api-access-khj6x\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx\" (UID: \"d9266c8c-0d14-41f6-93ac-aecddca1ff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" Nov 03 21:47:17.898794 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:17.898646 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" Nov 03 21:47:18.069775 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:18.069741 5041 generic.go:356] "Generic (PLEG): container finished" podID="1e56751c-ebda-4b44-8967-10b0e7ecbc81" containerID="3ce6087af1c70a9297e913f3b5b89c4b4af39ff751e74d6da144f99ec3977458" exitCode=0 Nov 03 21:47:18.070171 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:18.069835 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" event={"ID":"1e56751c-ebda-4b44-8967-10b0e7ecbc81","Type":"ContainerDied","Data":"3ce6087af1c70a9297e913f3b5b89c4b4af39ff751e74d6da144f99ec3977458"} Nov 03 21:47:18.214822 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:47:18.214785 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9266c8c_0d14_41f6_93ac_aecddca1ff89.slice/crio-0aab138e574112a2b2c58ff3969bcc740420540d4276c432a9dad08b815ed30d WatchSource:0}: Error finding container 0aab138e574112a2b2c58ff3969bcc740420540d4276c432a9dad08b815ed30d: Status 404 returned error can't find the container with id 0aab138e574112a2b2c58ff3969bcc740420540d4276c432a9dad08b815ed30d Nov 03 21:47:18.215043 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:18.215022 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx"] Nov 03 21:47:18.693461 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:18.693423 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr"] Nov 03 21:47:18.699726 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:18.699651 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" Nov 03 21:47:18.723112 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:18.723070 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr"] Nov 03 21:47:18.806522 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:18.806483 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19298766-3255-421f-98f8-f90cd92e55f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98a75a46-pjccr\" (UID: \"19298766-3255-421f-98f8-f90cd92e55f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" Nov 03 21:47:18.806767 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:18.806721 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19298766-3255-421f-98f8-f90cd92e55f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98a75a46-pjccr\" (UID: \"19298766-3255-421f-98f8-f90cd92e55f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" Nov 03 21:47:18.806872 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:18.806778 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2bwk\" (UniqueName: \"kubernetes.io/projected/19298766-3255-421f-98f8-f90cd92e55f3-kube-api-access-g2bwk\") pod \"maas-api-revoke-sub-98a75a46-pjccr\" (UID: \"19298766-3255-421f-98f8-f90cd92e55f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" Nov 03 21:47:18.917263 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:18.915770 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19298766-3255-421f-98f8-f90cd92e55f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98a75a46-pjccr\" (UID: \"19298766-3255-421f-98f8-f90cd92e55f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" Nov 03 21:47:18.917263 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:18.915867 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bwk\" (UniqueName: \"kubernetes.io/projected/19298766-3255-421f-98f8-f90cd92e55f3-kube-api-access-g2bwk\") pod \"maas-api-revoke-sub-98a75a46-pjccr\" (UID: \"19298766-3255-421f-98f8-f90cd92e55f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" Nov 03 21:47:18.917263 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:18.916630 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19298766-3255-421f-98f8-f90cd92e55f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98a75a46-pjccr\" (UID: \"19298766-3255-421f-98f8-f90cd92e55f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" Nov 03 21:47:18.917567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:18.917461 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19298766-3255-421f-98f8-f90cd92e55f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98a75a46-pjccr\" (UID: \"19298766-3255-421f-98f8-f90cd92e55f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" Nov 03 21:47:18.920569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:18.920541 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19298766-3255-421f-98f8-f90cd92e55f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98a75a46-pjccr\" (UID: \"19298766-3255-421f-98f8-f90cd92e55f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" Nov 03 21:47:18.937872 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:18.937843 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2bwk\" (UniqueName: \"kubernetes.io/projected/19298766-3255-421f-98f8-f90cd92e55f3-kube-api-access-g2bwk\") pod \"maas-api-revoke-sub-98a75a46-pjccr\" (UID: \"19298766-3255-421f-98f8-f90cd92e55f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" Nov 03 21:47:19.021004 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:19.020968 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" Nov 03 21:47:19.087917 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:19.086640 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" event={"ID":"d9266c8c-0d14-41f6-93ac-aecddca1ff89","Type":"ContainerStarted","Data":"7f79c34a4d0217dbfeaf3b85a472c32074f0c6ca7a3381167450a0d45df63b7c"} Nov 03 21:47:19.087917 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:19.086719 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" event={"ID":"d9266c8c-0d14-41f6-93ac-aecddca1ff89","Type":"ContainerStarted","Data":"0aab138e574112a2b2c58ff3969bcc740420540d4276c432a9dad08b815ed30d"} Nov 03 21:47:19.136936 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:19.136423 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" podStartSLOduration=2.136403361 podStartE2EDuration="2.136403361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:19.126129072 +0000 UTC m=+1054.633357425" watchObservedRunningTime="2026-11-03 21:47:19.136403361 +0000 UTC m=+1054.643631640" Nov 03 21:47:19.344398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:19.344370 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" Nov 03 21:47:19.349828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:19.349799 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr"] Nov 03 21:47:19.443552 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:19.443473 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e56751c-ebda-4b44-8967-10b0e7ecbc81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e56751c-ebda-4b44-8967-10b0e7ecbc81-maas-api-openshift-ca\") pod \"1e56751c-ebda-4b44-8967-10b0e7ecbc81\" (UID: \"1e56751c-ebda-4b44-8967-10b0e7ecbc81\") " Nov 03 21:47:19.443731 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:19.443592 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e56751c-ebda-4b44-8967-10b0e7ecbc81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e56751c-ebda-4b44-8967-10b0e7ecbc81-maas-api-xks-ca\") pod \"1e56751c-ebda-4b44-8967-10b0e7ecbc81\" (UID: \"1e56751c-ebda-4b44-8967-10b0e7ecbc81\") " Nov 03 21:47:19.443793 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:19.443748 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e56751c-ebda-4b44-8967-10b0e7ecbc81-kube-api-access-lxntg\" (UniqueName: \"kubernetes.io/projected/1e56751c-ebda-4b44-8967-10b0e7ecbc81-kube-api-access-lxntg\") pod \"1e56751c-ebda-4b44-8967-10b0e7ecbc81\" (UID: \"1e56751c-ebda-4b44-8967-10b0e7ecbc81\") " Nov 03 21:47:19.444686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:19.444631 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e56751c-ebda-4b44-8967-10b0e7ecbc81-maas-api-openshift-ca" pod "1e56751c-ebda-4b44-8967-10b0e7ecbc81" (UID: "1e56751c-ebda-4b44-8967-10b0e7ecbc81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:19.509376 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:19.509341 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e56751c-ebda-4b44-8967-10b0e7ecbc81-maas-api-xks-ca" pod "1e56751c-ebda-4b44-8967-10b0e7ecbc81" (UID: "1e56751c-ebda-4b44-8967-10b0e7ecbc81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:19.509872 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:19.509841 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e56751c-ebda-4b44-8967-10b0e7ecbc81-kube-api-access-lxntg" pod "1e56751c-ebda-4b44-8967-10b0e7ecbc81" (UID: "1e56751c-ebda-4b44-8967-10b0e7ecbc81"). InnerVolumeSpecName "kube-api-access-lxntg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:19.546232 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:19.546195 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e56751c-ebda-4b44-8967-10b0e7ecbc81-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:19.546232 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:19.546234 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e56751c-ebda-4b44-8967-10b0e7ecbc81-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:19.546474 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:19.546250 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxntg\" (UniqueName: \"kubernetes.io/projected/1e56751c-ebda-4b44-8967-10b0e7ecbc81-kube-api-access-lxntg\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:20.109690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:20.109581 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" event={"ID":"19298766-3255-421f-98f8-f90cd92e55f3","Type":"ContainerStarted","Data":"55e82961f73dcd4d75d535456e6fff75dde722ab588c78deb4ef060f75d51155"} Nov 03 21:47:20.110115 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:20.109690 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" event={"ID":"19298766-3255-421f-98f8-f90cd92e55f3","Type":"ContainerStarted","Data":"82667cbd75a5746ae6232ff6d6dbae990dce9d35ee26a075b83abbe97ad49292"} Nov 03 21:47:20.111488 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:20.111458 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" Nov 03 21:47:20.111651 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:20.111479 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484" event={"ID":"1e56751c-ebda-4b44-8967-10b0e7ecbc81","Type":"ContainerDied","Data":"b89359e6dd2ffbe16d5a06cf06bf1ea85f0a9f2b301664f09c0e45147e662ad8"} Nov 03 21:47:20.111651 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:20.111533 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b89359e6dd2ffbe16d5a06cf06bf1ea85f0a9f2b301664f09c0e45147e662ad8" Nov 03 21:47:20.113679 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:20.113606 5041 generic.go:356] "Generic (PLEG): container finished" podID="d9266c8c-0d14-41f6-93ac-aecddca1ff89" containerID="7f79c34a4d0217dbfeaf3b85a472c32074f0c6ca7a3381167450a0d45df63b7c" exitCode=0 Nov 03 21:47:20.113787 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:20.113739 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" event={"ID":"d9266c8c-0d14-41f6-93ac-aecddca1ff89","Type":"ContainerDied","Data":"7f79c34a4d0217dbfeaf3b85a472c32074f0c6ca7a3381167450a0d45df63b7c"} Nov 03 21:47:20.162892 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:20.162833 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" podStartSLOduration=2.162819331 podStartE2EDuration="2.162819331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:20.16133634 +0000 UTC m=+1055.668564640" watchObservedRunningTime="2026-11-03 21:47:20.162819331 +0000 UTC m=+1055.670047644" Nov 03 21:47:21.124712 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:21.124672 5041 generic.go:356] "Generic (PLEG): container finished" podID="19298766-3255-421f-98f8-f90cd92e55f3" containerID="55e82961f73dcd4d75d535456e6fff75dde722ab588c78deb4ef060f75d51155" exitCode=0 Nov 03 21:47:21.125145 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:21.124731 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" event={"ID":"19298766-3255-421f-98f8-f90cd92e55f3","Type":"ContainerDied","Data":"55e82961f73dcd4d75d535456e6fff75dde722ab588c78deb4ef060f75d51155"} Nov 03 21:47:21.340506 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:21.340471 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" Nov 03 21:47:21.647687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:21.643654 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9266c8c-0d14-41f6-93ac-aecddca1ff89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9266c8c-0d14-41f6-93ac-aecddca1ff89-maas-api-xks-ca\") pod \"d9266c8c-0d14-41f6-93ac-aecddca1ff89\" (UID: \"d9266c8c-0d14-41f6-93ac-aecddca1ff89\") " Nov 03 21:47:21.647687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:21.644014 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9266c8c-0d14-41f6-93ac-aecddca1ff89-kube-api-access-khj6x\" (UniqueName: \"kubernetes.io/projected/d9266c8c-0d14-41f6-93ac-aecddca1ff89-kube-api-access-khj6x\") pod \"d9266c8c-0d14-41f6-93ac-aecddca1ff89\" (UID: \"d9266c8c-0d14-41f6-93ac-aecddca1ff89\") " Nov 03 21:47:21.647687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:21.644124 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9266c8c-0d14-41f6-93ac-aecddca1ff89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9266c8c-0d14-41f6-93ac-aecddca1ff89-maas-api-openshift-ca\") pod \"d9266c8c-0d14-41f6-93ac-aecddca1ff89\" (UID: \"d9266c8c-0d14-41f6-93ac-aecddca1ff89\") " Nov 03 21:47:21.647687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:21.645358 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9266c8c-0d14-41f6-93ac-aecddca1ff89-maas-api-openshift-ca" pod "d9266c8c-0d14-41f6-93ac-aecddca1ff89" (UID: "d9266c8c-0d14-41f6-93ac-aecddca1ff89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:21.690686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:21.690181 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9266c8c-0d14-41f6-93ac-aecddca1ff89-kube-api-access-khj6x" pod "d9266c8c-0d14-41f6-93ac-aecddca1ff89" (UID: "d9266c8c-0d14-41f6-93ac-aecddca1ff89"). InnerVolumeSpecName "kube-api-access-khj6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:21.690686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:21.690683 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9266c8c-0d14-41f6-93ac-aecddca1ff89-maas-api-xks-ca" pod "d9266c8c-0d14-41f6-93ac-aecddca1ff89" (UID: "d9266c8c-0d14-41f6-93ac-aecddca1ff89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:21.748008 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:21.747555 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khj6x\" (UniqueName: \"kubernetes.io/projected/d9266c8c-0d14-41f6-93ac-aecddca1ff89-kube-api-access-khj6x\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:21.748008 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:21.747596 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9266c8c-0d14-41f6-93ac-aecddca1ff89-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:21.748008 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:21.747613 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9266c8c-0d14-41f6-93ac-aecddca1ff89-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:22.155466 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:22.155432 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" Nov 03 21:47:22.159872 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:22.159837 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx" event={"ID":"d9266c8c-0d14-41f6-93ac-aecddca1ff89","Type":"ContainerDied","Data":"0aab138e574112a2b2c58ff3969bcc740420540d4276c432a9dad08b815ed30d"} Nov 03 21:47:22.164855 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:22.164823 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aab138e574112a2b2c58ff3969bcc740420540d4276c432a9dad08b815ed30d" Nov 03 21:47:22.342794 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:22.342735 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs"] Nov 03 21:47:22.420348 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:22.419370 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk"] Nov 03 21:47:22.524686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:22.522632 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q"] Nov 03 21:47:23.625407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:23.624753 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:23.686916 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:23.641392 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.120:8000/ready\": context deadline exceeded" Nov 03 21:47:23.686916 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:23.672163 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:47:23.746752 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:23.731937 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:24.003748 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:23.930687 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:24.003748 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:23.932036 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:24.003748 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:23.932653 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 21:47:24.069694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.056580 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:24.220754 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.080199 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" Nov 03 21:47:24.400819 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.375694 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:24.432686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.432044 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19298766-3255-421f-98f8-f90cd92e55f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19298766-3255-421f-98f8-f90cd92e55f3-maas-api-xks-ca\") pod \"19298766-3255-421f-98f8-f90cd92e55f3\" (UID: \"19298766-3255-421f-98f8-f90cd92e55f3\") " Nov 03 21:47:24.467317 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.467273 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19298766-3255-421f-98f8-f90cd92e55f3-kube-api-access-g2bwk\" (UniqueName: \"kubernetes.io/projected/19298766-3255-421f-98f8-f90cd92e55f3-kube-api-access-g2bwk\") pod \"19298766-3255-421f-98f8-f90cd92e55f3\" (UID: \"19298766-3255-421f-98f8-f90cd92e55f3\") " Nov 03 21:47:24.493882 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.467639 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19298766-3255-421f-98f8-f90cd92e55f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19298766-3255-421f-98f8-f90cd92e55f3-maas-api-openshift-ca\") pod \"19298766-3255-421f-98f8-f90cd92e55f3\" (UID: \"19298766-3255-421f-98f8-f90cd92e55f3\") " Nov 03 21:47:24.495622 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.476990 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19298766-3255-421f-98f8-f90cd92e55f3-maas-api-openshift-ca" pod "19298766-3255-421f-98f8-f90cd92e55f3" (UID: "19298766-3255-421f-98f8-f90cd92e55f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:24.643274 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.643202 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.120:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:24.699929 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.699825 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:24.738278 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.738218 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:24.763685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.739766 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 21:47:24.792472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.792420 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19298766-3255-421f-98f8-f90cd92e55f3-maas-api-xks-ca" pod "19298766-3255-421f-98f8-f90cd92e55f3" (UID: "19298766-3255-421f-98f8-f90cd92e55f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:24.863722 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.863682 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19298766-3255-421f-98f8-f90cd92e55f3-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:24.864004 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.863981 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19298766-3255-421f-98f8-f90cd92e55f3-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:24.873688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.872609 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19298766-3255-421f-98f8-f90cd92e55f3-kube-api-access-g2bwk" pod "19298766-3255-421f-98f8-f90cd92e55f3" (UID: "19298766-3255-421f-98f8-f90cd92e55f3"). InnerVolumeSpecName "kube-api-access-g2bwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:24.939945 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.939891 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 21:47:25.097689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:24.945033 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:25.097689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:25.012775 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:25.097689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:25.013381 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2bwk\" (UniqueName: \"kubernetes.io/projected/19298766-3255-421f-98f8-f90cd92e55f3-kube-api-access-g2bwk\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:25.366748 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:25.361952 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:25.397885 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:25.374848 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 03 21:47:25.430056 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:25.430009 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:25.453369 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:25.453334 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:47:25.453631 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:25.453619 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:47:25.453773 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:25.453763 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:47:25.453876 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:25.453868 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:47:25.453980 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:25.453972 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:47:25.454078 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:25.454069 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:47:25.454179 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:25.454171 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:47:25.454281 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:25.454273 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:47:25.454380 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:25.454371 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:47:25.674750 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:25.659777 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.120:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:25.675540 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:25.675500 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" podUID="fa892a3f-4760-45ff-b00b-0840970540bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:26.421854 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:26.389260 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:26.421854 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:26.389699 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:47:26.429835 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:26.429782 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 21:47:26.430425 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:26.430391 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 21:47:26.430911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:26.430887 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:47:26.431300 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:26.431276 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:26.572766 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:26.526780 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" Nov 03 21:47:27.135689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:26.908221 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 21:47:27.136572 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:27.071723 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr" event={"ID":"19298766-3255-421f-98f8-f90cd92e55f3","Type":"ContainerDied","Data":"82667cbd75a5746ae6232ff6d6dbae990dce9d35ee26a075b83abbe97ad49292"} Nov 03 21:47:27.136802 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:27.136782 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82667cbd75a5746ae6232ff6d6dbae990dce9d35ee26a075b83abbe97ad49292" Nov 03 21:47:27.174689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:27.071782 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 21:47:27.176796 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:27.071832 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.120:8000/ready\": context deadline exceeded" Nov 03 21:47:27.358150 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:27.358092 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.120:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:27.446021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:27.435976 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:27.519690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:27.436839 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 21:47:27.519690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:27.452072 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:47:27.519690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:27.453159 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:27.519690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:27.490772 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 21:47:27.519690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:27.491113 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:47:27.800688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:27.754927 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:27.909278 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:27.909185 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:27.986827 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:27.986795 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4"] Nov 03 21:47:28.009688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:27.988222 5041 cpu_manager.go:414] "containerMap: removing container" podUID="1e56751c-ebda-4b44-8967-10b0e7ecbc81" containerName="revoke-keys" Nov 03 21:47:28.010008 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.009979 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e56751c-ebda-4b44-8967-10b0e7ecbc81" containerName="revoke-keys" Nov 03 21:47:28.010163 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.010149 5041 cpu_manager.go:414] "containerMap: removing container" podUID="d9266c8c-0d14-41f6-93ac-aecddca1ff89" containerName="revoke-keys" Nov 03 21:47:28.010262 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.010250 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9266c8c-0d14-41f6-93ac-aecddca1ff89" containerName="revoke-keys" Nov 03 21:47:28.010413 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.010402 5041 cpu_manager.go:414] "containerMap: removing container" podUID="19298766-3255-421f-98f8-f90cd92e55f3" containerName="revoke-keys" Nov 03 21:47:28.010500 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.010489 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19298766-3255-421f-98f8-f90cd92e55f3" containerName="revoke-keys" Nov 03 21:47:28.010878 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.010854 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9266c8c-0d14-41f6-93ac-aecddca1ff89" containerName="revoke-keys" Nov 03 21:47:28.011036 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.011021 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="19298766-3255-421f-98f8-f90cd92e55f3" containerName="revoke-keys" Nov 03 21:47:28.011140 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.011129 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e56751c-ebda-4b44-8967-10b0e7ecbc81" containerName="revoke-keys" Nov 03 21:47:28.187593 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.187460 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 21:47:28.283632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.283570 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.120:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:28.433684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.376791 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 21:47:28.789620 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.578168 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 03 21:47:28.789620 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.578202 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 21:47:28.789620 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.578223 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:47:28.827689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.578244 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 21:47:28.829683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.578262 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 21:47:28.829683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.578280 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:47:28.829683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.640174 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:47:28.829683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.829491 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" podUID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:28.829683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.683116 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:28.829683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.721133 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:47:28.830082 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.829707 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:47:28.830082 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.829724 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:28.830082 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.829756 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:28.894734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.872298 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:47:28.894734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.872360 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:28.938704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.895518 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-workload-certs\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:28.938704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.895591 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-credential-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:28.938704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.895624 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qrss\" (UniqueName: \"kubernetes.io/projected/8ad94b74-be79-48fd-847d-491eb6d5943a-kube-api-access-6qrss\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:28.938704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.895886 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-envoy\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:28.938704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.895950 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-data\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:28.938704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.896078 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-podinfo\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:28.938704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.896122 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-workload-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:28.938704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.896182 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad94b74-be79-48fd-847d-491eb6d5943a-istiod-ca-cert\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:28.938704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:28.896360 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-token\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.067572 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.067464 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-workload-certs\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.078832 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.078793 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-credential-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.083795 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.083755 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qrss\" (UniqueName: \"kubernetes.io/projected/8ad94b74-be79-48fd-847d-491eb6d5943a-kube-api-access-6qrss\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.084235 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.084211 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-envoy\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.087687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.084394 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-data\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.088023 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.087993 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-podinfo\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.128496 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.128300 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-workload-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.175361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.162679 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad94b74-be79-48fd-847d-491eb6d5943a-istiod-ca-cert\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.175361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.162851 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-token\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.175361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.129479 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-workload-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.175361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.166738 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-workload-certs\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.235089 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.235049 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad94b74-be79-48fd-847d-491eb6d5943a-istiod-ca-cert\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.262689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.259403 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-credential-socket\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.277910 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.277839 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:29.325744 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.300481 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.120:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:29.663182 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.663096 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:47:29.663389 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.405344 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:47:29.663522 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.663495 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:29.670301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.323325 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-data\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.765187 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.765137 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:29.776711 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.776648 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:29.805017 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.804940 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:29.846169 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.846122 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-envoy\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.889021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.871600 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 21:47:29.890451 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.871640 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:29.891338 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.878508 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 21:47:29.898476 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.898338 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:47:29.908343 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.907095 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-podinfo\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.911394 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.911367 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-token\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.912408 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.912386 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4"] Nov 03 21:47:29.912679 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.912649 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:47:29.937280 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.937191 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:29.972378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.972346 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:47:29.986124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.979998 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n"] Nov 03 21:47:29.991387 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.982876 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qrss\" (UniqueName: \"kubernetes.io/projected/8ad94b74-be79-48fd-847d-491eb6d5943a-kube-api-access-6qrss\") pod \"e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:29.999881 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:29.999852 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:47:30.001494 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.001472 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:47:30.002242 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.002041 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:47:30.003287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.003166 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.020083 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.020044 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:47:30.051428 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.051397 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:47:30.070040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.069993 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n"] Nov 03 21:47:30.089472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.089439 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2"] Nov 03 21:47:30.097248 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.097220 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2"] Nov 03 21:47:30.101903 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.098839 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" Nov 03 21:47:30.105769 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.105742 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh"] Nov 03 21:47:30.126921 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.125074 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:47:30.126921 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.125240 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:47:30.129369 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.129337 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" Nov 03 21:47:30.138180 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.138150 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh"] Nov 03 21:47:30.188158 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.188066 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt8gd\" (UniqueName: \"kubernetes.io/projected/5ba2a45a-0811-49ae-84db-468f7950fbb6-kube-api-access-dt8gd\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.188388 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.188366 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b91308d-036f-4fcb-9df5-1572f9821a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0da877d2-7lqb2\" (UID: \"1b91308d-036f-4fcb-9df5-1572f9821a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" Nov 03 21:47:30.188539 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.188520 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7j8l\" (UniqueName: \"kubernetes.io/projected/1b91308d-036f-4fcb-9df5-1572f9821a2c-kube-api-access-f7j8l\") pod \"maas-api-revoke-sub-0da877d2-7lqb2\" (UID: \"1b91308d-036f-4fcb-9df5-1572f9821a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" Nov 03 21:47:30.188862 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.188832 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb90896c-0b0c-4685-b2d4-207407cb4959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d33efaec-wrlsh\" (UID: \"eb90896c-0b0c-4685-b2d4-207407cb4959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" Nov 03 21:47:30.189115 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.189094 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbkz2\" (UniqueName: \"kubernetes.io/projected/eb90896c-0b0c-4685-b2d4-207407cb4959-kube-api-access-sbkz2\") pod \"maas-api-revoke-sub-d33efaec-wrlsh\" (UID: \"eb90896c-0b0c-4685-b2d4-207407cb4959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" Nov 03 21:47:30.189327 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.189304 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b91308d-036f-4fcb-9df5-1572f9821a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0da877d2-7lqb2\" (UID: \"1b91308d-036f-4fcb-9df5-1572f9821a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" Nov 03 21:47:30.189426 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.189400 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-workload-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.189530 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.189510 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-workload-certs\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.189643 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.189625 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-token\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.189919 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.189895 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb90896c-0b0c-4685-b2d4-207407cb4959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d33efaec-wrlsh\" (UID: \"eb90896c-0b0c-4685-b2d4-207407cb4959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" Nov 03 21:47:30.190010 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.189949 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-credential-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.190010 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.189997 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba2a45a-0811-49ae-84db-468f7950fbb6-istiod-ca-cert\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.190236 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.190113 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-podinfo\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.190236 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.190195 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-data\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.190483 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.190456 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-envoy\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.197980 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.197959 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6"] Nov 03 21:47:30.204768 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.204748 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6"] Nov 03 21:47:30.204971 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.204872 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" Nov 03 21:47:30.271876 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.271843 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:30.291646 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.291614 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-envoy\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.291785 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.291710 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt8gd\" (UniqueName: \"kubernetes.io/projected/5ba2a45a-0811-49ae-84db-468f7950fbb6-kube-api-access-dt8gd\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.291785 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.291743 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b91308d-036f-4fcb-9df5-1572f9821a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0da877d2-7lqb2\" (UID: \"1b91308d-036f-4fcb-9df5-1572f9821a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" Nov 03 21:47:30.291785 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.291772 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7j8l\" (UniqueName: \"kubernetes.io/projected/1b91308d-036f-4fcb-9df5-1572f9821a2c-kube-api-access-f7j8l\") pod \"maas-api-revoke-sub-0da877d2-7lqb2\" (UID: \"1b91308d-036f-4fcb-9df5-1572f9821a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" Nov 03 21:47:30.291941 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.291804 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb90896c-0b0c-4685-b2d4-207407cb4959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d33efaec-wrlsh\" (UID: \"eb90896c-0b0c-4685-b2d4-207407cb4959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" Nov 03 21:47:30.291941 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.291859 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb92ba06-6d13-4b60-a133-565d7c821595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92c4bdca-57zs6\" (UID: \"fb92ba06-6d13-4b60-a133-565d7c821595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" Nov 03 21:47:30.291941 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.291911 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbkz2\" (UniqueName: \"kubernetes.io/projected/eb90896c-0b0c-4685-b2d4-207407cb4959-kube-api-access-sbkz2\") pod \"maas-api-revoke-sub-d33efaec-wrlsh\" (UID: \"eb90896c-0b0c-4685-b2d4-207407cb4959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" Nov 03 21:47:30.292086 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.291963 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b91308d-036f-4fcb-9df5-1572f9821a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0da877d2-7lqb2\" (UID: \"1b91308d-036f-4fcb-9df5-1572f9821a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" Nov 03 21:47:30.292086 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.292054 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-workload-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.292791 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.292727 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b91308d-036f-4fcb-9df5-1572f9821a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0da877d2-7lqb2\" (UID: \"1b91308d-036f-4fcb-9df5-1572f9821a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" Nov 03 21:47:30.295194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.293406 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-workload-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.295194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.293530 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-workload-certs\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.295194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.293578 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-token\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.295194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.293617 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb90896c-0b0c-4685-b2d4-207407cb4959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d33efaec-wrlsh\" (UID: \"eb90896c-0b0c-4685-b2d4-207407cb4959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" Nov 03 21:47:30.295194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.293645 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-credential-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.295194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.293695 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba2a45a-0811-49ae-84db-468f7950fbb6-istiod-ca-cert\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.295194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.293733 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-podinfo\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.295194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.293785 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-data\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.295194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.293845 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb92ba06-6d13-4b60-a133-565d7c821595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92c4bdca-57zs6\" (UID: \"fb92ba06-6d13-4b60-a133-565d7c821595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" Nov 03 21:47:30.295194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.293900 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l52nw\" (UniqueName: \"kubernetes.io/projected/fb92ba06-6d13-4b60-a133-565d7c821595-kube-api-access-l52nw\") pod \"maas-api-revoke-sub-92c4bdca-57zs6\" (UID: \"fb92ba06-6d13-4b60-a133-565d7c821595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" Nov 03 21:47:30.295194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.293906 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-workload-certs\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.295194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.294408 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-credential-socket\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.295194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.294795 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba2a45a-0811-49ae-84db-468f7950fbb6-istiod-ca-cert\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.295194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.294926 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb90896c-0b0c-4685-b2d4-207407cb4959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d33efaec-wrlsh\" (UID: \"eb90896c-0b0c-4685-b2d4-207407cb4959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" Nov 03 21:47:30.295194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.295049 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-data\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.300248 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.300217 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-envoy\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.300341 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.300322 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb90896c-0b0c-4685-b2d4-207407cb4959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d33efaec-wrlsh\" (UID: \"eb90896c-0b0c-4685-b2d4-207407cb4959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" Nov 03 21:47:30.301168 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.301143 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-podinfo\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.302089 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.302062 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b91308d-036f-4fcb-9df5-1572f9821a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0da877d2-7lqb2\" (UID: \"1b91308d-036f-4fcb-9df5-1572f9821a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" Nov 03 21:47:30.344421 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.330988 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-token\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.346399 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.346341 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7j8l\" (UniqueName: \"kubernetes.io/projected/1b91308d-036f-4fcb-9df5-1572f9821a2c-kube-api-access-f7j8l\") pod \"maas-api-revoke-sub-0da877d2-7lqb2\" (UID: \"1b91308d-036f-4fcb-9df5-1572f9821a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" Nov 03 21:47:30.348218 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.348031 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt8gd\" (UniqueName: \"kubernetes.io/projected/5ba2a45a-0811-49ae-84db-468f7950fbb6-kube-api-access-dt8gd\") pod \"e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.350561 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.350525 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbkz2\" (UniqueName: \"kubernetes.io/projected/eb90896c-0b0c-4685-b2d4-207407cb4959-kube-api-access-sbkz2\") pod \"maas-api-revoke-sub-d33efaec-wrlsh\" (UID: \"eb90896c-0b0c-4685-b2d4-207407cb4959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" Nov 03 21:47:30.360031 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.356960 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:30.400291 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.399895 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb92ba06-6d13-4b60-a133-565d7c821595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92c4bdca-57zs6\" (UID: \"fb92ba06-6d13-4b60-a133-565d7c821595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" Nov 03 21:47:30.400291 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.399971 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l52nw\" (UniqueName: \"kubernetes.io/projected/fb92ba06-6d13-4b60-a133-565d7c821595-kube-api-access-l52nw\") pod \"maas-api-revoke-sub-92c4bdca-57zs6\" (UID: \"fb92ba06-6d13-4b60-a133-565d7c821595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" Nov 03 21:47:30.400291 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.400102 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb92ba06-6d13-4b60-a133-565d7c821595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92c4bdca-57zs6\" (UID: \"fb92ba06-6d13-4b60-a133-565d7c821595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" Nov 03 21:47:30.412415 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.402929 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb92ba06-6d13-4b60-a133-565d7c821595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92c4bdca-57zs6\" (UID: \"fb92ba06-6d13-4b60-a133-565d7c821595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" Nov 03 21:47:30.412415 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.404291 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb92ba06-6d13-4b60-a133-565d7c821595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92c4bdca-57zs6\" (UID: \"fb92ba06-6d13-4b60-a133-565d7c821595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" Nov 03 21:47:30.435822 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.433933 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l52nw\" (UniqueName: \"kubernetes.io/projected/fb92ba06-6d13-4b60-a133-565d7c821595-kube-api-access-l52nw\") pod \"maas-api-revoke-sub-92c4bdca-57zs6\" (UID: \"fb92ba06-6d13-4b60-a133-565d7c821595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" Nov 03 21:47:30.444675 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.443831 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" Nov 03 21:47:30.474268 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.474244 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" Nov 03 21:47:30.502903 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.502862 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr"] Nov 03 21:47:30.548072 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.541480 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" Nov 03 21:47:30.695482 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.695409 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4"] Nov 03 21:47:30.697026 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.696818 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:47:30.697026 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.696907 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:47:30.698162 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.698032 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:47:30.794030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.791574 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:47:30.794030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.791691 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:47:30.794030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.791740 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:47:30.794030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.791864 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n"] Nov 03 21:47:30.920869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:30.920819 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2"] Nov 03 21:47:31.043874 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:31.043008 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" event={"ID":"5ba2a45a-0811-49ae-84db-468f7950fbb6","Type":"ContainerStarted","Data":"1a23f6899f664aa83630f2f57214edf337bfe9c30801ab7695962debf8dd89fe"} Nov 03 21:47:31.043874 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:31.043062 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" event={"ID":"5ba2a45a-0811-49ae-84db-468f7950fbb6","Type":"ContainerStarted","Data":"185241c1dcb38732cd546a7a779683fbf7b41e0f92850935cdec3f854fe71054"} Nov 03 21:47:31.046588 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:31.045964 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" event={"ID":"8ad94b74-be79-48fd-847d-491eb6d5943a","Type":"ContainerStarted","Data":"90984b1f19703f54a6c67c35ab35f4cf8060884219525e7400acc6779b4d02d3"} Nov 03 21:47:31.046588 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:31.046018 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" event={"ID":"8ad94b74-be79-48fd-847d-491eb6d5943a","Type":"ContainerStarted","Data":"4d5b44da487dfdc4ce23175f280ccc1ed1b2849912a30b0e8e451b39b4a5462d"} Nov 03 21:47:31.047932 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:31.047901 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" event={"ID":"1b91308d-036f-4fcb-9df5-1572f9821a2c","Type":"ContainerStarted","Data":"70a1c3e0d67dbff75d0f3c01389366e9ca37dddfda54ed14bced83e26bbb65ad"} Nov 03 21:47:31.251552 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:31.251459 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" podStartSLOduration=6.2514357910000005 podStartE2EDuration="6.251435791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:31.223004791 +0000 UTC m=+1066.730233074" watchObservedRunningTime="2026-11-03 21:47:31.251435791 +0000 UTC m=+1066.758664072" Nov 03 21:47:31.317123 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:31.317039 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:31.330894 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:31.328888 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:47:31.330894 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:31.328949 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" podUID="8ad94b74-be79-48fd-847d-491eb6d5943a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 21:47:31.390584 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:31.390512 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" podStartSLOduration=6.390488006 podStartE2EDuration="6.390488006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:31.312344989 +0000 UTC m=+1066.819573270" watchObservedRunningTime="2026-11-03 21:47:31.390488006 +0000 UTC m=+1066.897716287" Nov 03 21:47:31.392954 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:31.392916 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:31.393111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:31.392999 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh"] Nov 03 21:47:31.438896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:31.434569 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 21:47:31.438896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:31.434632 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" podUID="5ba2a45a-0811-49ae-84db-468f7950fbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 21:47:31.438896 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:47:31.437465 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb90896c_0b0c_4685_b2d4_207407cb4959.slice/crio-a56d1e5c8ce99a2474ef05a940fd665dcbaf7e5e4a06ba8251315b15b4de5b6e WatchSource:0}: Error finding container a56d1e5c8ce99a2474ef05a940fd665dcbaf7e5e4a06ba8251315b15b4de5b6e: Status 404 returned error can't find the container with id a56d1e5c8ce99a2474ef05a940fd665dcbaf7e5e4a06ba8251315b15b4de5b6e Nov 03 21:47:31.471213 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:31.469117 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6"] Nov 03 21:47:32.074725 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.074641 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" event={"ID":"1b91308d-036f-4fcb-9df5-1572f9821a2c","Type":"ContainerStarted","Data":"2181969444fd8d1757b3c1fc627d9cfa40e433d65e7f97637e2f9803f8287224"} Nov 03 21:47:32.077845 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.077806 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" event={"ID":"fb92ba06-6d13-4b60-a133-565d7c821595","Type":"ContainerStarted","Data":"332c0e231e5c8a2ddd59ebe3bcecc3856d3735662a93c86468039c61c50c45f2"} Nov 03 21:47:32.078005 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.077869 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" event={"ID":"fb92ba06-6d13-4b60-a133-565d7c821595","Type":"ContainerStarted","Data":"997f2f3b6229d6fac566a6d99b15e60626f4055926d6780527b31b9cdc60db5b"} Nov 03 21:47:32.080778 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.080743 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" event={"ID":"eb90896c-0b0c-4685-b2d4-207407cb4959","Type":"ContainerStarted","Data":"7710cb7fd75d70a1178353af87156ef858f2bc30797e2cd0bb468d6fee5af142"} Nov 03 21:47:32.080944 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.080803 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" event={"ID":"eb90896c-0b0c-4685-b2d4-207407cb4959","Type":"ContainerStarted","Data":"a56d1e5c8ce99a2474ef05a940fd665dcbaf7e5e4a06ba8251315b15b4de5b6e"} Nov 03 21:47:32.175730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.175651 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" podStartSLOduration=3.175630699 podStartE2EDuration="3.175630699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:32.132594707 +0000 UTC m=+1067.639823014" watchObservedRunningTime="2026-11-03 21:47:32.175630699 +0000 UTC m=+1067.682858979" Nov 03 21:47:32.206410 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.206357 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" podStartSLOduration=3.206343693 podStartE2EDuration="3.206343693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:32.175000484 +0000 UTC m=+1067.682228764" watchObservedRunningTime="2026-11-03 21:47:32.206343693 +0000 UTC m=+1067.713572044" Nov 03 21:47:32.290685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.276451 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" podStartSLOduration=3.276426671 podStartE2EDuration="3.276426671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:32.231040283 +0000 UTC m=+1067.738268589" watchObservedRunningTime="2026-11-03 21:47:32.276426671 +0000 UTC m=+1067.783654954" Nov 03 21:47:32.290685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.284427 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:47:32.290685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.284486 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" podUID="8ad94b74-be79-48fd-847d-491eb6d5943a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 21:47:32.357841 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.357805 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 21:47:32.376918 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.358411 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" podUID="5ba2a45a-0811-49ae-84db-468f7950fbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 21:47:32.493934 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.493896 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b"] Nov 03 21:47:32.498487 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.498465 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" Nov 03 21:47:32.587212 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.587168 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-321bd940-2qf7b\" (UID: \"cfb2992a-df1d-4f5c-befd-b1d352bb79dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" Nov 03 21:47:32.587535 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.587515 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-321bd940-2qf7b\" (UID: \"cfb2992a-df1d-4f5c-befd-b1d352bb79dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" Nov 03 21:47:32.588043 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.588019 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k974d\" (UniqueName: \"kubernetes.io/projected/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-kube-api-access-k974d\") pod \"maas-api-revoke-sub-321bd940-2qf7b\" (UID: \"cfb2992a-df1d-4f5c-befd-b1d352bb79dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" Nov 03 21:47:32.593715 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.593688 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b"] Nov 03 21:47:32.719832 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.709211 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k974d\" (UniqueName: \"kubernetes.io/projected/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-kube-api-access-k974d\") pod \"maas-api-revoke-sub-321bd940-2qf7b\" (UID: \"cfb2992a-df1d-4f5c-befd-b1d352bb79dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" Nov 03 21:47:32.719832 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.709321 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-321bd940-2qf7b\" (UID: \"cfb2992a-df1d-4f5c-befd-b1d352bb79dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" Nov 03 21:47:32.719832 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.709429 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-321bd940-2qf7b\" (UID: \"cfb2992a-df1d-4f5c-befd-b1d352bb79dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" Nov 03 21:47:32.745747 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.744391 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-321bd940-2qf7b\" (UID: \"cfb2992a-df1d-4f5c-befd-b1d352bb79dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" Nov 03 21:47:32.866210 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.825527 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k974d\" (UniqueName: \"kubernetes.io/projected/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-kube-api-access-k974d\") pod \"maas-api-revoke-sub-321bd940-2qf7b\" (UID: \"cfb2992a-df1d-4f5c-befd-b1d352bb79dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" Nov 03 21:47:32.866210 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.857224 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-321bd940-2qf7b\" (UID: \"cfb2992a-df1d-4f5c-befd-b1d352bb79dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" Nov 03 21:47:32.893687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:32.891363 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" Nov 03 21:47:33.099470 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:33.098582 5041 generic.go:356] "Generic (PLEG): container finished" podID="1b91308d-036f-4fcb-9df5-1572f9821a2c" containerID="2181969444fd8d1757b3c1fc627d9cfa40e433d65e7f97637e2f9803f8287224" exitCode=0 Nov 03 21:47:33.099470 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:33.098701 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" event={"ID":"1b91308d-036f-4fcb-9df5-1572f9821a2c","Type":"ContainerDied","Data":"2181969444fd8d1757b3c1fc627d9cfa40e433d65e7f97637e2f9803f8287224"} Nov 03 21:47:33.102699 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:33.101510 5041 generic.go:356] "Generic (PLEG): container finished" podID="fb92ba06-6d13-4b60-a133-565d7c821595" containerID="332c0e231e5c8a2ddd59ebe3bcecc3856d3735662a93c86468039c61c50c45f2" exitCode=0 Nov 03 21:47:33.102699 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:33.101599 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" event={"ID":"fb92ba06-6d13-4b60-a133-565d7c821595","Type":"ContainerDied","Data":"332c0e231e5c8a2ddd59ebe3bcecc3856d3735662a93c86468039c61c50c45f2"} Nov 03 21:47:33.104153 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:33.103904 5041 generic.go:356] "Generic (PLEG): container finished" podID="eb90896c-0b0c-4685-b2d4-207407cb4959" containerID="7710cb7fd75d70a1178353af87156ef858f2bc30797e2cd0bb468d6fee5af142" exitCode=0 Nov 03 21:47:33.104153 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:33.104056 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" event={"ID":"eb90896c-0b0c-4685-b2d4-207407cb4959","Type":"ContainerDied","Data":"7710cb7fd75d70a1178353af87156ef858f2bc30797e2cd0bb468d6fee5af142"} Nov 03 21:47:33.282191 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:33.282147 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:47:33.282400 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:33.282211 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" podUID="8ad94b74-be79-48fd-847d-491eb6d5943a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 21:47:33.383841 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:33.382190 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 21:47:33.383841 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:33.382247 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" podUID="5ba2a45a-0811-49ae-84db-468f7950fbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 21:47:33.385948 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:47:33.385914 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb2992a_df1d_4f5c_befd_b1d352bb79dc.slice/crio-1aa88159d412924013558bcc83165e0751d8748bc100777dd25d650bcbd53ef2 WatchSource:0}: Error finding container 1aa88159d412924013558bcc83165e0751d8748bc100777dd25d650bcbd53ef2: Status 404 returned error can't find the container with id 1aa88159d412924013558bcc83165e0751d8748bc100777dd25d650bcbd53ef2 Nov 03 21:47:33.387309 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:33.387283 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b"] Nov 03 21:47:34.201682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:34.198778 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" event={"ID":"cfb2992a-df1d-4f5c-befd-b1d352bb79dc","Type":"ContainerStarted","Data":"113a7f4bd8f4b1d9651b8f6b520edae1403f05d70118a93ff10506a61c7eec5a"} Nov 03 21:47:34.201682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:34.198876 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" event={"ID":"cfb2992a-df1d-4f5c-befd-b1d352bb79dc","Type":"ContainerStarted","Data":"1aa88159d412924013558bcc83165e0751d8748bc100777dd25d650bcbd53ef2"} Nov 03 21:47:34.284622 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:34.283963 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:47:34.284622 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:34.284036 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" podUID="8ad94b74-be79-48fd-847d-491eb6d5943a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 21:47:34.323094 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:34.321474 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" podStartSLOduration=2.321454675 podStartE2EDuration="2.321454675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:34.320048015 +0000 UTC m=+1069.827276297" watchObservedRunningTime="2026-11-03 21:47:34.321454675 +0000 UTC m=+1069.828682956" Nov 03 21:47:34.366704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:34.363167 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 21:47:34.366704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:34.363229 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" podUID="5ba2a45a-0811-49ae-84db-468f7950fbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 21:47:35.098050 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.097921 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" Nov 03 21:47:35.199201 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.199167 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb90896c-0b0c-4685-b2d4-207407cb4959-kube-api-access-sbkz2\" (UniqueName: \"kubernetes.io/projected/eb90896c-0b0c-4685-b2d4-207407cb4959-kube-api-access-sbkz2\") pod \"eb90896c-0b0c-4685-b2d4-207407cb4959\" (UID: \"eb90896c-0b0c-4685-b2d4-207407cb4959\") " Nov 03 21:47:35.199614 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.199578 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb90896c-0b0c-4685-b2d4-207407cb4959-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb90896c-0b0c-4685-b2d4-207407cb4959-maas-api-openshift-ca\") pod \"eb90896c-0b0c-4685-b2d4-207407cb4959\" (UID: \"eb90896c-0b0c-4685-b2d4-207407cb4959\") " Nov 03 21:47:35.199768 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.199653 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb90896c-0b0c-4685-b2d4-207407cb4959-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb90896c-0b0c-4685-b2d4-207407cb4959-maas-api-xks-ca\") pod \"eb90896c-0b0c-4685-b2d4-207407cb4959\" (UID: \"eb90896c-0b0c-4685-b2d4-207407cb4959\") " Nov 03 21:47:35.200276 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.200247 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb90896c-0b0c-4685-b2d4-207407cb4959-maas-api-openshift-ca" pod "eb90896c-0b0c-4685-b2d4-207407cb4959" (UID: "eb90896c-0b0c-4685-b2d4-207407cb4959"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:35.201724 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.200544 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb90896c-0b0c-4685-b2d4-207407cb4959-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:35.202573 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.202515 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb90896c-0b0c-4685-b2d4-207407cb4959-kube-api-access-sbkz2" pod "eb90896c-0b0c-4685-b2d4-207407cb4959" (UID: "eb90896c-0b0c-4685-b2d4-207407cb4959"). InnerVolumeSpecName "kube-api-access-sbkz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:35.202646 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.202621 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb90896c-0b0c-4685-b2d4-207407cb4959-maas-api-xks-ca" pod "eb90896c-0b0c-4685-b2d4-207407cb4959" (UID: "eb90896c-0b0c-4685-b2d4-207407cb4959"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:35.277685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.277214 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:47:35.288688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.287751 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" podUID="8ad94b74-be79-48fd-847d-491eb6d5943a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 21:47:35.299347 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.299181 5041 generic.go:356] "Generic (PLEG): container finished" podID="cfb2992a-df1d-4f5c-befd-b1d352bb79dc" containerID="113a7f4bd8f4b1d9651b8f6b520edae1403f05d70118a93ff10506a61c7eec5a" exitCode=0 Nov 03 21:47:35.303552 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.301223 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" event={"ID":"cfb2992a-df1d-4f5c-befd-b1d352bb79dc","Type":"ContainerDied","Data":"113a7f4bd8f4b1d9651b8f6b520edae1403f05d70118a93ff10506a61c7eec5a"} Nov 03 21:47:35.303552 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.302764 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb90896c-0b0c-4685-b2d4-207407cb4959-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:35.303552 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.302790 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbkz2\" (UniqueName: \"kubernetes.io/projected/eb90896c-0b0c-4685-b2d4-207407cb4959-kube-api-access-sbkz2\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:35.304070 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.304039 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" event={"ID":"eb90896c-0b0c-4685-b2d4-207407cb4959","Type":"ContainerDied","Data":"a56d1e5c8ce99a2474ef05a940fd665dcbaf7e5e4a06ba8251315b15b4de5b6e"} Nov 03 21:47:35.304166 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.304099 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a56d1e5c8ce99a2474ef05a940fd665dcbaf7e5e4a06ba8251315b15b4de5b6e" Nov 03 21:47:35.304228 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.304164 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh" Nov 03 21:47:35.388769 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.388602 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 21:47:35.388769 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.388686 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" podUID="5ba2a45a-0811-49ae-84db-468f7950fbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 21:47:35.492119 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.492092 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" Nov 03 21:47:35.496822 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.496797 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" Nov 03 21:47:35.504298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.504269 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" podUID="7526f386-4f83-49e7-a3e0-80bb549fec6c" containerName="payload-processing" containerID="cri-o://07566408ab0372e0d8235a22dd8ac83e1755ea66c7640050bf921d7d37ff93c8" gracePeriod=25 Nov 03 21:47:35.607049 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.607012 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb92ba06-6d13-4b60-a133-565d7c821595-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb92ba06-6d13-4b60-a133-565d7c821595-maas-api-xks-ca\") pod \"fb92ba06-6d13-4b60-a133-565d7c821595\" (UID: \"fb92ba06-6d13-4b60-a133-565d7c821595\") " Nov 03 21:47:35.607200 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.607085 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb92ba06-6d13-4b60-a133-565d7c821595-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb92ba06-6d13-4b60-a133-565d7c821595-maas-api-openshift-ca\") pod \"fb92ba06-6d13-4b60-a133-565d7c821595\" (UID: \"fb92ba06-6d13-4b60-a133-565d7c821595\") " Nov 03 21:47:35.607200 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.607174 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb92ba06-6d13-4b60-a133-565d7c821595-kube-api-access-l52nw\" (UniqueName: \"kubernetes.io/projected/fb92ba06-6d13-4b60-a133-565d7c821595-kube-api-access-l52nw\") pod \"fb92ba06-6d13-4b60-a133-565d7c821595\" (UID: \"fb92ba06-6d13-4b60-a133-565d7c821595\") " Nov 03 21:47:35.607350 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.607331 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b91308d-036f-4fcb-9df5-1572f9821a2c-kube-api-access-f7j8l\" (UniqueName: \"kubernetes.io/projected/1b91308d-036f-4fcb-9df5-1572f9821a2c-kube-api-access-f7j8l\") pod \"1b91308d-036f-4fcb-9df5-1572f9821a2c\" (UID: \"1b91308d-036f-4fcb-9df5-1572f9821a2c\") " Nov 03 21:47:35.607401 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.607384 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b91308d-036f-4fcb-9df5-1572f9821a2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b91308d-036f-4fcb-9df5-1572f9821a2c-maas-api-xks-ca\") pod \"1b91308d-036f-4fcb-9df5-1572f9821a2c\" (UID: \"1b91308d-036f-4fcb-9df5-1572f9821a2c\") " Nov 03 21:47:35.607448 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.607437 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b91308d-036f-4fcb-9df5-1572f9821a2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b91308d-036f-4fcb-9df5-1572f9821a2c-maas-api-openshift-ca\") pod \"1b91308d-036f-4fcb-9df5-1572f9821a2c\" (UID: \"1b91308d-036f-4fcb-9df5-1572f9821a2c\") " Nov 03 21:47:35.619710 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.607650 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb92ba06-6d13-4b60-a133-565d7c821595-maas-api-openshift-ca" pod "fb92ba06-6d13-4b60-a133-565d7c821595" (UID: "fb92ba06-6d13-4b60-a133-565d7c821595"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:35.619710 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.608313 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb92ba06-6d13-4b60-a133-565d7c821595-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:35.643782 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.628339 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b91308d-036f-4fcb-9df5-1572f9821a2c-maas-api-openshift-ca" pod "1b91308d-036f-4fcb-9df5-1572f9821a2c" (UID: "1b91308d-036f-4fcb-9df5-1572f9821a2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:35.643782 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.632032 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b91308d-036f-4fcb-9df5-1572f9821a2c-maas-api-xks-ca" pod "1b91308d-036f-4fcb-9df5-1572f9821a2c" (UID: "1b91308d-036f-4fcb-9df5-1572f9821a2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:35.643782 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.632264 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b91308d-036f-4fcb-9df5-1572f9821a2c-kube-api-access-f7j8l" pod "1b91308d-036f-4fcb-9df5-1572f9821a2c" (UID: "1b91308d-036f-4fcb-9df5-1572f9821a2c"). InnerVolumeSpecName "kube-api-access-f7j8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:35.649833 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.649776 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb92ba06-6d13-4b60-a133-565d7c821595-kube-api-access-l52nw" pod "fb92ba06-6d13-4b60-a133-565d7c821595" (UID: "fb92ba06-6d13-4b60-a133-565d7c821595"). InnerVolumeSpecName "kube-api-access-l52nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:35.650099 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.650076 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb92ba06-6d13-4b60-a133-565d7c821595-maas-api-xks-ca" pod "fb92ba06-6d13-4b60-a133-565d7c821595" (UID: "fb92ba06-6d13-4b60-a133-565d7c821595"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:35.731688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.731192 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7j8l\" (UniqueName: \"kubernetes.io/projected/1b91308d-036f-4fcb-9df5-1572f9821a2c-kube-api-access-f7j8l\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:35.731688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.731239 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b91308d-036f-4fcb-9df5-1572f9821a2c-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:35.731688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.731255 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b91308d-036f-4fcb-9df5-1572f9821a2c-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:35.731688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.731269 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb92ba06-6d13-4b60-a133-565d7c821595-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:35.731688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:35.731285 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l52nw\" (UniqueName: \"kubernetes.io/projected/fb92ba06-6d13-4b60-a133-565d7c821595-kube-api-access-l52nw\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:36.189776 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.189736 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" Nov 03 21:47:36.283233 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.278057 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:47:36.283233 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.278118 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" podUID="8ad94b74-be79-48fd-847d-491eb6d5943a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 21:47:36.295851 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.295793 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7526f386-4f83-49e7-a3e0-80bb549fec6c-kube-api-access-bbnvt\" (UniqueName: \"kubernetes.io/projected/7526f386-4f83-49e7-a3e0-80bb549fec6c-kube-api-access-bbnvt\") pod \"7526f386-4f83-49e7-a3e0-80bb549fec6c\" (UID: \"7526f386-4f83-49e7-a3e0-80bb549fec6c\") " Nov 03 21:47:36.296415 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.296052 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7526f386-4f83-49e7-a3e0-80bb549fec6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7526f386-4f83-49e7-a3e0-80bb549fec6c-plugins-config-volume\") pod \"7526f386-4f83-49e7-a3e0-80bb549fec6c\" (UID: \"7526f386-4f83-49e7-a3e0-80bb549fec6c\") " Nov 03 21:47:36.297644 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.297611 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7526f386-4f83-49e7-a3e0-80bb549fec6c-plugins-config-volume" pod "7526f386-4f83-49e7-a3e0-80bb549fec6c" (UID: "7526f386-4f83-49e7-a3e0-80bb549fec6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:36.298642 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.298617 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7526f386-4f83-49e7-a3e0-80bb549fec6c-kube-api-access-bbnvt" pod "7526f386-4f83-49e7-a3e0-80bb549fec6c" (UID: "7526f386-4f83-49e7-a3e0-80bb549fec6c"). InnerVolumeSpecName "kube-api-access-bbnvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:36.394471 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.393024 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 21:47:36.394471 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.393089 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" podUID="5ba2a45a-0811-49ae-84db-468f7950fbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 21:47:36.398964 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.398773 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbnvt\" (UniqueName: \"kubernetes.io/projected/7526f386-4f83-49e7-a3e0-80bb549fec6c-kube-api-access-bbnvt\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:36.398964 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.398805 5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7526f386-4f83-49e7-a3e0-80bb549fec6c-plugins-config-volume\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:36.407566 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.407527 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" event={"ID":"1b91308d-036f-4fcb-9df5-1572f9821a2c","Type":"ContainerDied","Data":"70a1c3e0d67dbff75d0f3c01389366e9ca37dddfda54ed14bced83e26bbb65ad"} Nov 03 21:47:36.415828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.407594 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70a1c3e0d67dbff75d0f3c01389366e9ca37dddfda54ed14bced83e26bbb65ad" Nov 03 21:47:36.415828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.407719 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2" Nov 03 21:47:36.487114 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.486732 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" event={"ID":"fb92ba06-6d13-4b60-a133-565d7c821595","Type":"ContainerDied","Data":"997f2f3b6229d6fac566a6d99b15e60626f4055926d6780527b31b9cdc60db5b"} Nov 03 21:47:36.487114 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.486796 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="997f2f3b6229d6fac566a6d99b15e60626f4055926d6780527b31b9cdc60db5b" Nov 03 21:47:36.487114 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.486856 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6" Nov 03 21:47:36.491567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.491445 5041 generic.go:356] "Generic (PLEG): container finished" podID="7526f386-4f83-49e7-a3e0-80bb549fec6c" containerID="07566408ab0372e0d8235a22dd8ac83e1755ea66c7640050bf921d7d37ff93c8" exitCode=0 Nov 03 21:47:36.491567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.491517 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" Nov 03 21:47:36.491771 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.491566 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" event={"ID":"7526f386-4f83-49e7-a3e0-80bb549fec6c","Type":"ContainerDied","Data":"07566408ab0372e0d8235a22dd8ac83e1755ea66c7640050bf921d7d37ff93c8"} Nov 03 21:47:36.491771 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.491613 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr" event={"ID":"7526f386-4f83-49e7-a3e0-80bb549fec6c","Type":"ContainerDied","Data":"126e1520864d8319a6efabd719ae32a2967c9d5da2ce68e06cb05cf16a98b14a"} Nov 03 21:47:36.491771 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.491634 5041 scope.go:122] "RemoveContainer" containerID="07566408ab0372e0d8235a22dd8ac83e1755ea66c7640050bf921d7d37ff93c8" Nov 03 21:47:36.544907 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.520854 5041 scope.go:122] "RemoveContainer" containerID="07566408ab0372e0d8235a22dd8ac83e1755ea66c7640050bf921d7d37ff93c8" Nov 03 21:47:36.544907 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:47:36.522164 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07566408ab0372e0d8235a22dd8ac83e1755ea66c7640050bf921d7d37ff93c8\": container with ID starting with 07566408ab0372e0d8235a22dd8ac83e1755ea66c7640050bf921d7d37ff93c8 not found: ID does not exist" containerID="07566408ab0372e0d8235a22dd8ac83e1755ea66c7640050bf921d7d37ff93c8" Nov 03 21:47:36.544907 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.522205 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07566408ab0372e0d8235a22dd8ac83e1755ea66c7640050bf921d7d37ff93c8"} err="failed to get container status \"07566408ab0372e0d8235a22dd8ac83e1755ea66c7640050bf921d7d37ff93c8\": rpc error: code = NotFound desc = could not find container \"07566408ab0372e0d8235a22dd8ac83e1755ea66c7640050bf921d7d37ff93c8\": container with ID starting with 07566408ab0372e0d8235a22dd8ac83e1755ea66c7640050bf921d7d37ff93c8 not found: ID does not exist" Nov 03 21:47:36.702259 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.702225 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr"] Nov 03 21:47:36.712403 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:36.711955 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-dd5620-575c97994b-452fr"] Nov 03 21:47:37.139682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.130941 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" Nov 03 21:47:37.185681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.185547 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-kube-api-access-k974d\" (UniqueName: \"kubernetes.io/projected/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-kube-api-access-k974d\") pod \"cfb2992a-df1d-4f5c-befd-b1d352bb79dc\" (UID: \"cfb2992a-df1d-4f5c-befd-b1d352bb79dc\") " Nov 03 21:47:37.185681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.185628 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-maas-api-xks-ca\") pod \"cfb2992a-df1d-4f5c-befd-b1d352bb79dc\" (UID: \"cfb2992a-df1d-4f5c-befd-b1d352bb79dc\") " Nov 03 21:47:37.185887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.185703 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-maas-api-openshift-ca\") pod \"cfb2992a-df1d-4f5c-befd-b1d352bb79dc\" (UID: \"cfb2992a-df1d-4f5c-befd-b1d352bb79dc\") " Nov 03 21:47:37.186645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.186620 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-maas-api-openshift-ca" pod "cfb2992a-df1d-4f5c-befd-b1d352bb79dc" (UID: "cfb2992a-df1d-4f5c-befd-b1d352bb79dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:37.188398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.188361 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-kube-api-access-k974d" pod "cfb2992a-df1d-4f5c-befd-b1d352bb79dc" (UID: "cfb2992a-df1d-4f5c-befd-b1d352bb79dc"). InnerVolumeSpecName "kube-api-access-k974d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:37.189802 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.189768 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-maas-api-xks-ca" pod "cfb2992a-df1d-4f5c-befd-b1d352bb79dc" (UID: "cfb2992a-df1d-4f5c-befd-b1d352bb79dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:37.276934 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.276893 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:47:37.277218 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.277184 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" podUID="8ad94b74-be79-48fd-847d-491eb6d5943a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 21:47:37.289688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.289404 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k974d\" (UniqueName: \"kubernetes.io/projected/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-kube-api-access-k974d\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:37.289688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.289436 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:37.289688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.289446 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb2992a-df1d-4f5c-befd-b1d352bb79dc-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:37.366444 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.366133 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 21:47:37.366444 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.366200 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" podUID="5ba2a45a-0811-49ae-84db-468f7950fbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 21:47:37.387900 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.387864 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255"] Nov 03 21:47:37.388805 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.388785 5041 cpu_manager.go:414] "containerMap: removing container" podUID="cfb2992a-df1d-4f5c-befd-b1d352bb79dc" containerName="revoke-keys" Nov 03 21:47:37.388805 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.388804 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb2992a-df1d-4f5c-befd-b1d352bb79dc" containerName="revoke-keys" Nov 03 21:47:37.388931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.388823 5041 cpu_manager.go:414] "containerMap: removing container" podUID="1b91308d-036f-4fcb-9df5-1572f9821a2c" containerName="revoke-keys" Nov 03 21:47:37.388931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.388829 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b91308d-036f-4fcb-9df5-1572f9821a2c" containerName="revoke-keys" Nov 03 21:47:37.388931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.388843 5041 cpu_manager.go:414] "containerMap: removing container" podUID="eb90896c-0b0c-4685-b2d4-207407cb4959" containerName="revoke-keys" Nov 03 21:47:37.388931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.388849 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb90896c-0b0c-4685-b2d4-207407cb4959" containerName="revoke-keys" Nov 03 21:47:37.388931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.388865 5041 cpu_manager.go:414] "containerMap: removing container" podUID="fb92ba06-6d13-4b60-a133-565d7c821595" containerName="revoke-keys" Nov 03 21:47:37.388931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.388869 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb92ba06-6d13-4b60-a133-565d7c821595" containerName="revoke-keys" Nov 03 21:47:37.388931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.388883 5041 cpu_manager.go:414] "containerMap: removing container" podUID="7526f386-4f83-49e7-a3e0-80bb549fec6c" containerName="payload-processing" Nov 03 21:47:37.388931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.388887 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7526f386-4f83-49e7-a3e0-80bb549fec6c" containerName="payload-processing" Nov 03 21:47:37.389166 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.388988 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b91308d-036f-4fcb-9df5-1572f9821a2c" containerName="revoke-keys" Nov 03 21:47:37.389166 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.389000 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb90896c-0b0c-4685-b2d4-207407cb4959" containerName="revoke-keys" Nov 03 21:47:37.389166 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.389008 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb92ba06-6d13-4b60-a133-565d7c821595" containerName="revoke-keys" Nov 03 21:47:37.389166 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.389014 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7526f386-4f83-49e7-a3e0-80bb549fec6c" containerName="payload-processing" Nov 03 21:47:37.389166 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.389025 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb2992a-df1d-4f5c-befd-b1d352bb79dc" containerName="revoke-keys" Nov 03 21:47:37.393246 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.393224 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255" Nov 03 21:47:37.438683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.421231 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255"] Nov 03 21:47:37.493279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.493238 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a96e07b-e342-4628-a091-00528b5d6d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14e905ad-l4255\" (UID: \"5a96e07b-e342-4628-a091-00528b5d6d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255" Nov 03 21:47:37.493483 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.493436 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a96e07b-e342-4628-a091-00528b5d6d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14e905ad-l4255\" (UID: \"5a96e07b-e342-4628-a091-00528b5d6d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255" Nov 03 21:47:37.493573 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.493514 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2kp2\" (UniqueName: \"kubernetes.io/projected/5a96e07b-e342-4628-a091-00528b5d6d1c-kube-api-access-x2kp2\") pod \"maas-api-revoke-sub-14e905ad-l4255\" (UID: \"5a96e07b-e342-4628-a091-00528b5d6d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255" Nov 03 21:47:37.515622 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.515578 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" event={"ID":"cfb2992a-df1d-4f5c-befd-b1d352bb79dc","Type":"ContainerDied","Data":"1aa88159d412924013558bcc83165e0751d8748bc100777dd25d650bcbd53ef2"} Nov 03 21:47:37.515967 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.515944 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aa88159d412924013558bcc83165e0751d8748bc100777dd25d650bcbd53ef2" Nov 03 21:47:37.516261 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.516211 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b" Nov 03 21:47:37.594933 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.594892 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a96e07b-e342-4628-a091-00528b5d6d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14e905ad-l4255\" (UID: \"5a96e07b-e342-4628-a091-00528b5d6d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255" Nov 03 21:47:37.595151 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.595036 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a96e07b-e342-4628-a091-00528b5d6d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14e905ad-l4255\" (UID: \"5a96e07b-e342-4628-a091-00528b5d6d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255" Nov 03 21:47:37.595151 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.595102 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2kp2\" (UniqueName: \"kubernetes.io/projected/5a96e07b-e342-4628-a091-00528b5d6d1c-kube-api-access-x2kp2\") pod \"maas-api-revoke-sub-14e905ad-l4255\" (UID: \"5a96e07b-e342-4628-a091-00528b5d6d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255" Nov 03 21:47:37.595968 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.595935 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a96e07b-e342-4628-a091-00528b5d6d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14e905ad-l4255\" (UID: \"5a96e07b-e342-4628-a091-00528b5d6d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255" Nov 03 21:47:37.597482 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.597457 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a96e07b-e342-4628-a091-00528b5d6d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14e905ad-l4255\" (UID: \"5a96e07b-e342-4628-a091-00528b5d6d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255" Nov 03 21:47:37.641762 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.638485 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2kp2\" (UniqueName: \"kubernetes.io/projected/5a96e07b-e342-4628-a091-00528b5d6d1c-kube-api-access-x2kp2\") pod \"maas-api-revoke-sub-14e905ad-l4255\" (UID: \"5a96e07b-e342-4628-a091-00528b5d6d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255" Nov 03 21:47:37.741999 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:37.741956 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255" Nov 03 21:47:38.075944 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:47:38.075882 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a96e07b_e342_4628_a091_00528b5d6d1c.slice/crio-ac710e90887138f114af518f1aae9f9b25330c655fcca13f9a6f2312b94a067f WatchSource:0}: Error finding container ac710e90887138f114af518f1aae9f9b25330c655fcca13f9a6f2312b94a067f: Status 404 returned error can't find the container with id ac710e90887138f114af518f1aae9f9b25330c655fcca13f9a6f2312b94a067f Nov 03 21:47:38.076313 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:38.076288 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255"] Nov 03 21:47:38.273041 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:38.272992 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:47:38.273571 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:38.273057 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" podUID="8ad94b74-be79-48fd-847d-491eb6d5943a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 21:47:38.358758 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:38.358639 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 21:47:38.358758 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:38.358743 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" podUID="5ba2a45a-0811-49ae-84db-468f7950fbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 21:47:38.540127 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:38.540079 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255" event={"ID":"5a96e07b-e342-4628-a091-00528b5d6d1c","Type":"ContainerStarted","Data":"ac710e90887138f114af518f1aae9f9b25330c655fcca13f9a6f2312b94a067f"} Nov 03 21:47:38.631230 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:38.631194 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7526f386-4f83-49e7-a3e0-80bb549fec6c" path="/var/lib/kubelet/pods/7526f386-4f83-49e7-a3e0-80bb549fec6c/volumes" Nov 03 21:47:39.111452 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:39.111410 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4"] Nov 03 21:47:39.278121 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:39.278082 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:39.279278 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:39.279247 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:39.281859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:39.280606 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:47:39.364523 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:39.364426 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:39.365755 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:39.365729 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:39.366592 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:39.366570 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:47:39.553524 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:39.553489 5041 generic.go:356] "Generic (PLEG): container finished" podID="5a96e07b-e342-4628-a091-00528b5d6d1c" containerID="72a0a9e334c3748df5e8de15f723eda542a364d013137ffdbebb357a4f1175c8" exitCode=0 Nov 03 21:47:39.553759 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:39.553579 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255" event={"ID":"5a96e07b-e342-4628-a091-00528b5d6d1c","Type":"ContainerDied","Data":"72a0a9e334c3748df5e8de15f723eda542a364d013137ffdbebb357a4f1175c8"} Nov 03 21:47:40.741770 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:40.741742 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255" Nov 03 21:47:40.785716 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:47:40.782150 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:47:40.860112 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:40.860033 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a96e07b-e342-4628-a091-00528b5d6d1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a96e07b-e342-4628-a091-00528b5d6d1c-maas-api-xks-ca\") pod \"5a96e07b-e342-4628-a091-00528b5d6d1c\" (UID: \"5a96e07b-e342-4628-a091-00528b5d6d1c\") " Nov 03 21:47:40.860387 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:40.860372 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a96e07b-e342-4628-a091-00528b5d6d1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a96e07b-e342-4628-a091-00528b5d6d1c-maas-api-openshift-ca\") pod \"5a96e07b-e342-4628-a091-00528b5d6d1c\" (UID: \"5a96e07b-e342-4628-a091-00528b5d6d1c\") " Nov 03 21:47:40.860437 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:40.860403 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a96e07b-e342-4628-a091-00528b5d6d1c-kube-api-access-x2kp2\" (UniqueName: \"kubernetes.io/projected/5a96e07b-e342-4628-a091-00528b5d6d1c-kube-api-access-x2kp2\") pod \"5a96e07b-e342-4628-a091-00528b5d6d1c\" (UID: \"5a96e07b-e342-4628-a091-00528b5d6d1c\") " Nov 03 21:47:40.860838 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:40.860804 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a96e07b-e342-4628-a091-00528b5d6d1c-maas-api-openshift-ca" pod "5a96e07b-e342-4628-a091-00528b5d6d1c" (UID: "5a96e07b-e342-4628-a091-00528b5d6d1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:40.866137 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:40.861767 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a96e07b-e342-4628-a091-00528b5d6d1c-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:40.885712 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:40.885656 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a96e07b-e342-4628-a091-00528b5d6d1c-maas-api-xks-ca" pod "5a96e07b-e342-4628-a091-00528b5d6d1c" (UID: "5a96e07b-e342-4628-a091-00528b5d6d1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:40.885949 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:40.885918 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a96e07b-e342-4628-a091-00528b5d6d1c-kube-api-access-x2kp2" pod "5a96e07b-e342-4628-a091-00528b5d6d1c" (UID: "5a96e07b-e342-4628-a091-00528b5d6d1c"). InnerVolumeSpecName "kube-api-access-x2kp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:40.963442 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:40.963405 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2kp2\" (UniqueName: \"kubernetes.io/projected/5a96e07b-e342-4628-a091-00528b5d6d1c-kube-api-access-x2kp2\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:40.963442 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:40.963441 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a96e07b-e342-4628-a091-00528b5d6d1c-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:41.576862 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:41.576825 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255" Nov 03 21:47:41.576862 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:41.576835 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255" event={"ID":"5a96e07b-e342-4628-a091-00528b5d6d1c","Type":"ContainerDied","Data":"ac710e90887138f114af518f1aae9f9b25330c655fcca13f9a6f2312b94a067f"} Nov 03 21:47:41.577085 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:41.576914 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac710e90887138f114af518f1aae9f9b25330c655fcca13f9a6f2312b94a067f" Nov 03 21:47:43.916702 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:43.916642 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n"] Nov 03 21:47:43.917417 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:43.917404 5041 cpu_manager.go:414] "containerMap: removing container" podUID="5a96e07b-e342-4628-a091-00528b5d6d1c" containerName="revoke-keys" Nov 03 21:47:43.917490 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:43.917419 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a96e07b-e342-4628-a091-00528b5d6d1c" containerName="revoke-keys" Nov 03 21:47:43.917552 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:43.917541 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a96e07b-e342-4628-a091-00528b5d6d1c" containerName="revoke-keys" Nov 03 21:47:43.922776 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:43.922751 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" Nov 03 21:47:43.940479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:43.940443 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n"] Nov 03 21:47:43.945655 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:43.945625 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:47:43.945842 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:43.945695 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:47:44.015301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.015251 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2157382b-e7ba-44bf-9814-124cac52c539-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n\" (UID: \"2157382b-e7ba-44bf-9814-124cac52c539\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" Nov 03 21:47:44.015495 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.015419 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2157382b-e7ba-44bf-9814-124cac52c539-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n\" (UID: \"2157382b-e7ba-44bf-9814-124cac52c539\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" Nov 03 21:47:44.015621 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.015595 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rglb2\" (UniqueName: \"kubernetes.io/projected/2157382b-e7ba-44bf-9814-124cac52c539-kube-api-access-rglb2\") pod \"maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n\" (UID: \"2157382b-e7ba-44bf-9814-124cac52c539\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" Nov 03 21:47:44.117233 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.117200 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rglb2\" (UniqueName: \"kubernetes.io/projected/2157382b-e7ba-44bf-9814-124cac52c539-kube-api-access-rglb2\") pod \"maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n\" (UID: \"2157382b-e7ba-44bf-9814-124cac52c539\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" Nov 03 21:47:44.117453 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.117406 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2157382b-e7ba-44bf-9814-124cac52c539-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n\" (UID: \"2157382b-e7ba-44bf-9814-124cac52c539\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" Nov 03 21:47:44.117526 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.117483 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2157382b-e7ba-44bf-9814-124cac52c539-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n\" (UID: \"2157382b-e7ba-44bf-9814-124cac52c539\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" Nov 03 21:47:44.118209 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.118180 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2157382b-e7ba-44bf-9814-124cac52c539-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n\" (UID: \"2157382b-e7ba-44bf-9814-124cac52c539\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" Nov 03 21:47:44.119887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.119868 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2157382b-e7ba-44bf-9814-124cac52c539-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n\" (UID: \"2157382b-e7ba-44bf-9814-124cac52c539\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" Nov 03 21:47:44.137672 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.137633 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rglb2\" (UniqueName: \"kubernetes.io/projected/2157382b-e7ba-44bf-9814-124cac52c539-kube-api-access-rglb2\") pod \"maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n\" (UID: \"2157382b-e7ba-44bf-9814-124cac52c539\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" Nov 03 21:47:44.244204 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.244165 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" Nov 03 21:47:44.557865 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:47:44.557814 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2157382b_e7ba_44bf_9814_124cac52c539.slice/crio-b608d2dbf504005d6d97f3899ebd99a8992eaa232a11529a07f7f1ebedbd5eae WatchSource:0}: Error finding container b608d2dbf504005d6d97f3899ebd99a8992eaa232a11529a07f7f1ebedbd5eae: Status 404 returned error can't find the container with id b608d2dbf504005d6d97f3899ebd99a8992eaa232a11529a07f7f1ebedbd5eae Nov 03 21:47:44.560129 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.560102 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n"] Nov 03 21:47:44.628367 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.628326 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" containerID="cri-o://faa7e30f9a287f45a4856e7d2032e5f055df733a1d3df587d6f983f9e7387551" gracePeriod=38 Nov 03 21:47:44.632053 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.632020 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" event={"ID":"2157382b-e7ba-44bf-9814-124cac52c539","Type":"ContainerStarted","Data":"b608d2dbf504005d6d97f3899ebd99a8992eaa232a11529a07f7f1ebedbd5eae"} Nov 03 21:47:44.726514 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.726480 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": dial tcp 10.132.0.109:8000: connect: connection refused" Nov 03 21:47:44.888623 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.888585 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9"] Nov 03 21:47:44.913382 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.913343 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" containerID="cri-o://719b99d06895604c9a764bb33269df18fbbda2ca7e372df484501a7cdee215d6" gracePeriod=38 Nov 03 21:47:44.977267 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:44.977211 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" containerID="cri-o://22ae285ab38147b029a3cbe3b9ca94ee4537f4a36a7170afa4239c55038e77e9" gracePeriod=38 Nov 03 21:47:45.170816 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.170409 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:47:45.266312 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.264868 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049539c3-9a7f-4b23-b307-9e634705772a-kube-api-access-vkxk2\" (UniqueName: \"kubernetes.io/projected/049539c3-9a7f-4b23-b307-9e634705772a-kube-api-access-vkxk2\") pod \"049539c3-9a7f-4b23-b307-9e634705772a\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " Nov 03 21:47:45.266312 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.265071 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-home\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-home\") pod \"049539c3-9a7f-4b23-b307-9e634705772a\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " Nov 03 21:47:45.266312 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.265114 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-tmp-dir\") pod \"049539c3-9a7f-4b23-b307-9e634705772a\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " Nov 03 21:47:45.266312 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.265152 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-dshm\") pod \"049539c3-9a7f-4b23-b307-9e634705772a\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " Nov 03 21:47:45.266312 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.265359 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-model-cache\") pod \"049539c3-9a7f-4b23-b307-9e634705772a\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " Nov 03 21:47:45.266312 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.265398 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/049539c3-9a7f-4b23-b307-9e634705772a-tls-certs\" (UniqueName: \"kubernetes.io/secret/049539c3-9a7f-4b23-b307-9e634705772a-tls-certs\") pod \"049539c3-9a7f-4b23-b307-9e634705772a\" (UID: \"049539c3-9a7f-4b23-b307-9e634705772a\") " Nov 03 21:47:45.266717 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.266510 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-tmp-dir" pod "049539c3-9a7f-4b23-b307-9e634705772a" (UID: "049539c3-9a7f-4b23-b307-9e634705772a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:45.267184 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.267100 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-home" pod "049539c3-9a7f-4b23-b307-9e634705772a" (UID: "049539c3-9a7f-4b23-b307-9e634705772a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:45.267184 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.267159 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-model-cache" pod "049539c3-9a7f-4b23-b307-9e634705772a" (UID: "049539c3-9a7f-4b23-b307-9e634705772a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:45.271398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.271064 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-dshm" pod "049539c3-9a7f-4b23-b307-9e634705772a" (UID: "049539c3-9a7f-4b23-b307-9e634705772a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:45.280816 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.280782 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/049539c3-9a7f-4b23-b307-9e634705772a-tls-certs" pod "049539c3-9a7f-4b23-b307-9e634705772a" (UID: "049539c3-9a7f-4b23-b307-9e634705772a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:45.315679 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.310070 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049539c3-9a7f-4b23-b307-9e634705772a-kube-api-access-vkxk2" pod "049539c3-9a7f-4b23-b307-9e634705772a" (UID: "049539c3-9a7f-4b23-b307-9e634705772a"). InnerVolumeSpecName "kube-api-access-vkxk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:45.369860 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.369834 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkxk2\" (UniqueName: \"kubernetes.io/projected/049539c3-9a7f-4b23-b307-9e634705772a-kube-api-access-vkxk2\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.369968 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.369865 5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-home\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.369968 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.369881 5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-tmp-dir\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.369968 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.369893 5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-dshm\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.369968 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.369905 5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/049539c3-9a7f-4b23-b307-9e634705772a-model-cache\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.369968 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.369917 5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049539c3-9a7f-4b23-b307-9e634705772a-tls-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.417735 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.417638 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:47:45.473459 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.470512 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-dshm\") pod \"f09eb21d-9777-437b-a2a1-efce30d9d145\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " Nov 03 21:47:45.473459 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.470719 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f09eb21d-9777-437b-a2a1-efce30d9d145-tls-certs\" (UniqueName: \"kubernetes.io/secret/f09eb21d-9777-437b-a2a1-efce30d9d145-tls-certs\") pod \"f09eb21d-9777-437b-a2a1-efce30d9d145\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " Nov 03 21:47:45.473459 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.470814 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f09eb21d-9777-437b-a2a1-efce30d9d145-kube-api-access-s45m7\" (UniqueName: \"kubernetes.io/projected/f09eb21d-9777-437b-a2a1-efce30d9d145-kube-api-access-s45m7\") pod \"f09eb21d-9777-437b-a2a1-efce30d9d145\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " Nov 03 21:47:45.473459 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.470898 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-model-cache\") pod \"f09eb21d-9777-437b-a2a1-efce30d9d145\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " Nov 03 21:47:45.473459 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.471044 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-tmp-dir\") pod \"f09eb21d-9777-437b-a2a1-efce30d9d145\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " Nov 03 21:47:45.473459 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.471140 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-home\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-home\") pod \"f09eb21d-9777-437b-a2a1-efce30d9d145\" (UID: \"f09eb21d-9777-437b-a2a1-efce30d9d145\") " Nov 03 21:47:45.496687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.488914 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-home" pod "f09eb21d-9777-437b-a2a1-efce30d9d145" (UID: "f09eb21d-9777-437b-a2a1-efce30d9d145"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:45.496687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.489149 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-model-cache" pod "f09eb21d-9777-437b-a2a1-efce30d9d145" (UID: "f09eb21d-9777-437b-a2a1-efce30d9d145"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:45.496687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.490823 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-dshm" pod "f09eb21d-9777-437b-a2a1-efce30d9d145" (UID: "f09eb21d-9777-437b-a2a1-efce30d9d145"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:45.496687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.491794 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:47:45.557694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.512534 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-tmp-dir" pod "f09eb21d-9777-437b-a2a1-efce30d9d145" (UID: "f09eb21d-9777-437b-a2a1-efce30d9d145"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:45.744738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.722266 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-dshm\") pod \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " Nov 03 21:47:45.744738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.732934 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-home\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-home\") pod \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " Nov 03 21:47:45.744738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.737119 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-home" pod "2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" (UID: "2fd6761c-c37e-4d69-83a6-7ea3ee5e9118"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:45.746732 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.746678 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-dshm" pod "2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" (UID: "2fd6761c-c37e-4d69-83a6-7ea3ee5e9118"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:45.753892 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.753861 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-tls-certs\") pod \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " Nov 03 21:47:45.754090 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.754073 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-model-cache\") pod \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " Nov 03 21:47:45.754206 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.754142 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-tmp-dir\") pod \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " Nov 03 21:47:45.754265 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.754234 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-kube-api-access-hd46h\" (UniqueName: \"kubernetes.io/projected/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-kube-api-access-hd46h\") pod \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\" (UID: \"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118\") " Nov 03 21:47:45.755002 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.754978 5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-dshm\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.755115 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.755004 5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-model-cache\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.755115 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.755019 5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-home\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.755115 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.755031 5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-tmp-dir\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.755115 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.755042 5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-home\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.755115 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.755052 5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f09eb21d-9777-437b-a2a1-efce30d9d145-dshm\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.755739 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.755505 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f09eb21d-9777-437b-a2a1-efce30d9d145-tls-certs" pod "f09eb21d-9777-437b-a2a1-efce30d9d145" (UID: "f09eb21d-9777-437b-a2a1-efce30d9d145"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:45.756180 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.755893 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-model-cache" pod "2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" (UID: "2fd6761c-c37e-4d69-83a6-7ea3ee5e9118"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:45.760969 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.760471 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-kube-api-access-hd46h" pod "2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" (UID: "2fd6761c-c37e-4d69-83a6-7ea3ee5e9118"). InnerVolumeSpecName "kube-api-access-hd46h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:45.760969 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.760722 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-tmp-dir" pod "2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" (UID: "2fd6761c-c37e-4d69-83a6-7ea3ee5e9118"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:45.761244 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.761167 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-tls-certs" pod "2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" (UID: "2fd6761c-c37e-4d69-83a6-7ea3ee5e9118"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:45.765820 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.764670 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" event={"ID":"2157382b-e7ba-44bf-9814-124cac52c539","Type":"ContainerStarted","Data":"27878dbe06d0d2c03b121a5ac2534098b7f52b21d860fab1e532c8b6cb132d7f"} Nov 03 21:47:45.769715 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.769678 5041 generic.go:356] "Generic (PLEG): container finished" podID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerID="719b99d06895604c9a764bb33269df18fbbda2ca7e372df484501a7cdee215d6" exitCode=0 Nov 03 21:47:45.770432 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.769919 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" Nov 03 21:47:45.770432 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.769939 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" event={"ID":"f09eb21d-9777-437b-a2a1-efce30d9d145","Type":"ContainerDied","Data":"719b99d06895604c9a764bb33269df18fbbda2ca7e372df484501a7cdee215d6"} Nov 03 21:47:45.770432 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.770264 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk" event={"ID":"f09eb21d-9777-437b-a2a1-efce30d9d145","Type":"ContainerDied","Data":"aacc20229b9a72889cfbe394b2b0262774f7af85ca93b3caa9f23717718edaf1"} Nov 03 21:47:45.770432 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.770288 5041 scope.go:122] "RemoveContainer" containerID="719b99d06895604c9a764bb33269df18fbbda2ca7e372df484501a7cdee215d6" Nov 03 21:47:45.784950 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.784873 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09eb21d-9777-437b-a2a1-efce30d9d145-kube-api-access-s45m7" pod "f09eb21d-9777-437b-a2a1-efce30d9d145" (UID: "f09eb21d-9777-437b-a2a1-efce30d9d145"). InnerVolumeSpecName "kube-api-access-s45m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:45.789468 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.787716 5041 generic.go:356] "Generic (PLEG): container finished" podID="049539c3-9a7f-4b23-b307-9e634705772a" containerID="faa7e30f9a287f45a4856e7d2032e5f055df733a1d3df587d6f983f9e7387551" exitCode=0 Nov 03 21:47:45.789468 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.787815 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" event={"ID":"049539c3-9a7f-4b23-b307-9e634705772a","Type":"ContainerDied","Data":"faa7e30f9a287f45a4856e7d2032e5f055df733a1d3df587d6f983f9e7387551"} Nov 03 21:47:45.789468 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.787848 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" event={"ID":"049539c3-9a7f-4b23-b307-9e634705772a","Type":"ContainerDied","Data":"68f993a50b84d007639cf0ebedb922cc2560f3c67b39d1cb7846f5e253a4cf92"} Nov 03 21:47:45.789468 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.787909 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs" Nov 03 21:47:45.804852 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.802341 5041 generic.go:356] "Generic (PLEG): container finished" podID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerID="22ae285ab38147b029a3cbe3b9ca94ee4537f4a36a7170afa4239c55038e77e9" exitCode=0 Nov 03 21:47:45.804852 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.802613 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" event={"ID":"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118","Type":"ContainerDied","Data":"22ae285ab38147b029a3cbe3b9ca94ee4537f4a36a7170afa4239c55038e77e9"} Nov 03 21:47:45.804852 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.802649 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" event={"ID":"2fd6761c-c37e-4d69-83a6-7ea3ee5e9118","Type":"ContainerDied","Data":"8c5705b93b696818de2b1c016e53799e6733a0f830055513c4a98ae0004f2d1e"} Nov 03 21:47:45.804852 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.802788 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q" Nov 03 21:47:45.820328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.820303 5041 scope.go:122] "RemoveContainer" containerID="719b99d06895604c9a764bb33269df18fbbda2ca7e372df484501a7cdee215d6" Nov 03 21:47:45.820917 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:47:45.820892 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"719b99d06895604c9a764bb33269df18fbbda2ca7e372df484501a7cdee215d6\": container with ID starting with 719b99d06895604c9a764bb33269df18fbbda2ca7e372df484501a7cdee215d6 not found: ID does not exist" containerID="719b99d06895604c9a764bb33269df18fbbda2ca7e372df484501a7cdee215d6" Nov 03 21:47:45.821028 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.820924 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"719b99d06895604c9a764bb33269df18fbbda2ca7e372df484501a7cdee215d6"} err="failed to get container status \"719b99d06895604c9a764bb33269df18fbbda2ca7e372df484501a7cdee215d6\": rpc error: code = NotFound desc = could not find container \"719b99d06895604c9a764bb33269df18fbbda2ca7e372df484501a7cdee215d6\": container with ID starting with 719b99d06895604c9a764bb33269df18fbbda2ca7e372df484501a7cdee215d6 not found: ID does not exist" Nov 03 21:47:45.821028 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.820944 5041 scope.go:122] "RemoveContainer" containerID="faa7e30f9a287f45a4856e7d2032e5f055df733a1d3df587d6f983f9e7387551" Nov 03 21:47:45.826230 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.821915 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" podStartSLOduration=2.821898139 podStartE2EDuration="2.821898139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:45.814801284 +0000 UTC m=+1081.322029566" watchObservedRunningTime="2026-11-03 21:47:45.821898139 +0000 UTC m=+1081.329126446" Nov 03 21:47:45.848972 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.843720 5041 scope.go:122] "RemoveContainer" containerID="faa7e30f9a287f45a4856e7d2032e5f055df733a1d3df587d6f983f9e7387551" Nov 03 21:47:45.848972 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:47:45.844248 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa7e30f9a287f45a4856e7d2032e5f055df733a1d3df587d6f983f9e7387551\": container with ID starting with faa7e30f9a287f45a4856e7d2032e5f055df733a1d3df587d6f983f9e7387551 not found: ID does not exist" containerID="faa7e30f9a287f45a4856e7d2032e5f055df733a1d3df587d6f983f9e7387551" Nov 03 21:47:45.848972 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.844289 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa7e30f9a287f45a4856e7d2032e5f055df733a1d3df587d6f983f9e7387551"} err="failed to get container status \"faa7e30f9a287f45a4856e7d2032e5f055df733a1d3df587d6f983f9e7387551\": rpc error: code = NotFound desc = could not find container \"faa7e30f9a287f45a4856e7d2032e5f055df733a1d3df587d6f983f9e7387551\": container with ID starting with faa7e30f9a287f45a4856e7d2032e5f055df733a1d3df587d6f983f9e7387551 not found: ID does not exist" Nov 03 21:47:45.848972 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.844315 5041 scope.go:122] "RemoveContainer" containerID="22ae285ab38147b029a3cbe3b9ca94ee4537f4a36a7170afa4239c55038e77e9" Nov 03 21:47:45.860366 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.860336 5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f09eb21d-9777-437b-a2a1-efce30d9d145-tls-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.860519 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.860373 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s45m7\" (UniqueName: \"kubernetes.io/projected/f09eb21d-9777-437b-a2a1-efce30d9d145-kube-api-access-s45m7\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.860519 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.860388 5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-tls-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.860519 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.860403 5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-model-cache\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.860519 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.860415 5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-tmp-dir\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.860519 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.860427 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd46h\" (UniqueName: \"kubernetes.io/projected/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118-kube-api-access-hd46h\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:45.863377 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.863355 5041 scope.go:122] "RemoveContainer" containerID="22ae285ab38147b029a3cbe3b9ca94ee4537f4a36a7170afa4239c55038e77e9" Nov 03 21:47:45.863716 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:47:45.863696 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ae285ab38147b029a3cbe3b9ca94ee4537f4a36a7170afa4239c55038e77e9\": container with ID starting with 22ae285ab38147b029a3cbe3b9ca94ee4537f4a36a7170afa4239c55038e77e9 not found: ID does not exist" containerID="22ae285ab38147b029a3cbe3b9ca94ee4537f4a36a7170afa4239c55038e77e9" Nov 03 21:47:45.863801 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.863723 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ae285ab38147b029a3cbe3b9ca94ee4537f4a36a7170afa4239c55038e77e9"} err="failed to get container status \"22ae285ab38147b029a3cbe3b9ca94ee4537f4a36a7170afa4239c55038e77e9\": rpc error: code = NotFound desc = could not find container \"22ae285ab38147b029a3cbe3b9ca94ee4537f4a36a7170afa4239c55038e77e9\": container with ID starting with 22ae285ab38147b029a3cbe3b9ca94ee4537f4a36a7170afa4239c55038e77e9 not found: ID does not exist" Nov 03 21:47:45.905351 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.905320 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9"] Nov 03 21:47:45.906102 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.906087 5041 cpu_manager.go:414] "containerMap: removing container" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" Nov 03 21:47:45.906145 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.906103 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" Nov 03 21:47:45.906145 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.906124 5041 cpu_manager.go:414] "containerMap: removing container" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" Nov 03 21:47:45.906145 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.906130 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" Nov 03 21:47:45.906145 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.906145 5041 cpu_manager.go:414] "containerMap: removing container" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" Nov 03 21:47:45.906278 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.906150 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" Nov 03 21:47:45.906278 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.906255 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" containerName="main" Nov 03 21:47:45.906278 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.906270 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="049539c3-9a7f-4b23-b307-9e634705772a" containerName="main" Nov 03 21:47:45.906368 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.906280 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" containerName="main" Nov 03 21:47:45.911986 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.911964 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs"] Nov 03 21:47:45.912109 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.912045 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9" Nov 03 21:47:45.923106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.923071 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/distinct-2-w3-dd5620-kserve-6b7c5dbd8d-4rdrs"] Nov 03 21:47:45.933625 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.933597 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9"] Nov 03 21:47:45.950974 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.950941 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q"] Nov 03 21:47:45.957075 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:45.957048 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/simulator-w3-dd5620-kserve-55969759c-ftz8q"] Nov 03 21:47:46.064367 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.064285 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9\" (UID: \"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9" Nov 03 21:47:46.064367 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.064324 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5bw7\" (UniqueName: \"kubernetes.io/projected/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-kube-api-access-n5bw7\") pod \"maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9\" (UID: \"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9" Nov 03 21:47:46.064367 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.064357 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9\" (UID: \"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9" Nov 03 21:47:46.122546 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.122512 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk"] Nov 03 21:47:46.132388 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.132354 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-dd5620/premium-w3-dd5620-kserve-85bfdb5f88-wbgmk"] Nov 03 21:47:46.166174 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.166136 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9\" (UID: \"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9" Nov 03 21:47:46.166424 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.166406 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9\" (UID: \"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9" Nov 03 21:47:46.166500 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.166435 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5bw7\" (UniqueName: \"kubernetes.io/projected/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-kube-api-access-n5bw7\") pod \"maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9\" (UID: \"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9" Nov 03 21:47:46.166808 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.166787 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9\" (UID: \"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9" Nov 03 21:47:46.168792 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.168770 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9\" (UID: \"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9" Nov 03 21:47:46.188818 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.188784 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5bw7\" (UniqueName: \"kubernetes.io/projected/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-kube-api-access-n5bw7\") pod \"maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9\" (UID: \"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9" Nov 03 21:47:46.235761 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.235730 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9" Nov 03 21:47:46.547984 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:47:46.547953 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fd0876_cfb0_4dd0_9e7f_0c3e49bfe651.slice/crio-eb3e99dfefad948d7a3550800cb059e137e46b8f666f5d7cd5fccc581908f01a WatchSource:0}: Error finding container eb3e99dfefad948d7a3550800cb059e137e46b8f666f5d7cd5fccc581908f01a: Status 404 returned error can't find the container with id eb3e99dfefad948d7a3550800cb059e137e46b8f666f5d7cd5fccc581908f01a Nov 03 21:47:46.550543 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.550516 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9"] Nov 03 21:47:46.625479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.625422 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049539c3-9a7f-4b23-b307-9e634705772a" path="/var/lib/kubelet/pods/049539c3-9a7f-4b23-b307-9e634705772a/volumes" Nov 03 21:47:46.625984 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.625968 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd6761c-c37e-4d69-83a6-7ea3ee5e9118" path="/var/lib/kubelet/pods/2fd6761c-c37e-4d69-83a6-7ea3ee5e9118/volumes" Nov 03 21:47:46.626403 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.626390 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f09eb21d-9777-437b-a2a1-efce30d9d145" path="/var/lib/kubelet/pods/f09eb21d-9777-437b-a2a1-efce30d9d145/volumes" Nov 03 21:47:46.816184 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.816152 5041 generic.go:356] "Generic (PLEG): container finished" podID="2157382b-e7ba-44bf-9814-124cac52c539" containerID="27878dbe06d0d2c03b121a5ac2534098b7f52b21d860fab1e532c8b6cb132d7f" exitCode=0 Nov 03 21:47:46.816387 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.816283 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" event={"ID":"2157382b-e7ba-44bf-9814-124cac52c539","Type":"ContainerDied","Data":"27878dbe06d0d2c03b121a5ac2534098b7f52b21d860fab1e532c8b6cb132d7f"} Nov 03 21:47:46.818081 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.818049 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9" event={"ID":"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651","Type":"ContainerStarted","Data":"eb3e99dfefad948d7a3550800cb059e137e46b8f666f5d7cd5fccc581908f01a"} Nov 03 21:47:46.836373 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.836346 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt"] Nov 03 21:47:46.843175 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.843149 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt" Nov 03 21:47:46.864412 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.864382 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt"] Nov 03 21:47:46.980862 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.980829 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cb49f77-sdqdt\" (UID: \"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt" Nov 03 21:47:46.981044 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.980956 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cb49f77-sdqdt\" (UID: \"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt" Nov 03 21:47:46.981044 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:46.981019 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8j4v\" (UniqueName: \"kubernetes.io/projected/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-kube-api-access-g8j4v\") pod \"maas-api-revoke-sub-8cb49f77-sdqdt\" (UID: \"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt" Nov 03 21:47:47.082969 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:47.082872 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8j4v\" (UniqueName: \"kubernetes.io/projected/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-kube-api-access-g8j4v\") pod \"maas-api-revoke-sub-8cb49f77-sdqdt\" (UID: \"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt" Nov 03 21:47:47.083421 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:47.083193 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cb49f77-sdqdt\" (UID: \"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt" Nov 03 21:47:47.083421 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:47.083292 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cb49f77-sdqdt\" (UID: \"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt" Nov 03 21:47:47.083954 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:47.083927 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cb49f77-sdqdt\" (UID: \"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt" Nov 03 21:47:47.085645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:47.085609 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cb49f77-sdqdt\" (UID: \"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt" Nov 03 21:47:47.103409 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:47.103383 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8j4v\" (UniqueName: \"kubernetes.io/projected/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-kube-api-access-g8j4v\") pod \"maas-api-revoke-sub-8cb49f77-sdqdt\" (UID: \"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt" Nov 03 21:47:47.166404 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:47.166374 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt" Nov 03 21:47:47.476858 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:47.476826 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt"] Nov 03 21:47:47.477192 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:47:47.477161 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a88155b_e3ff_4308_94ee_d8cea2d3ed6a.slice/crio-b5bede310696b9e4e897fb33ddbd23119e545d63ebd080d312bf77d6e34f2fb7 WatchSource:0}: Error finding container b5bede310696b9e4e897fb33ddbd23119e545d63ebd080d312bf77d6e34f2fb7: Status 404 returned error can't find the container with id b5bede310696b9e4e897fb33ddbd23119e545d63ebd080d312bf77d6e34f2fb7 Nov 03 21:47:47.854507 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:47.854467 5041 generic.go:356] "Generic (PLEG): container finished" podID="a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651" containerID="5866060ddbf7f3a07faf44cabe0a62466a74dd92cb97a8c7393b163372d23dd7" exitCode=0 Nov 03 21:47:47.854717 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:47.854543 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9" event={"ID":"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651","Type":"ContainerDied","Data":"5866060ddbf7f3a07faf44cabe0a62466a74dd92cb97a8c7393b163372d23dd7"} Nov 03 21:47:47.856531 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:47.856502 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt" event={"ID":"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a","Type":"ContainerStarted","Data":"b5bede310696b9e4e897fb33ddbd23119e545d63ebd080d312bf77d6e34f2fb7"} Nov 03 21:47:47.997490 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:47.997462 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" Nov 03 21:47:48.099193 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:48.099105 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2157382b-e7ba-44bf-9814-124cac52c539-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2157382b-e7ba-44bf-9814-124cac52c539-maas-api-openshift-ca\") pod \"2157382b-e7ba-44bf-9814-124cac52c539\" (UID: \"2157382b-e7ba-44bf-9814-124cac52c539\") " Nov 03 21:47:48.099707 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:48.099204 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2157382b-e7ba-44bf-9814-124cac52c539-kube-api-access-rglb2\" (UniqueName: \"kubernetes.io/projected/2157382b-e7ba-44bf-9814-124cac52c539-kube-api-access-rglb2\") pod \"2157382b-e7ba-44bf-9814-124cac52c539\" (UID: \"2157382b-e7ba-44bf-9814-124cac52c539\") " Nov 03 21:47:48.099707 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:48.099435 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2157382b-e7ba-44bf-9814-124cac52c539-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2157382b-e7ba-44bf-9814-124cac52c539-maas-api-xks-ca\") pod \"2157382b-e7ba-44bf-9814-124cac52c539\" (UID: \"2157382b-e7ba-44bf-9814-124cac52c539\") " Nov 03 21:47:48.099707 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:48.099522 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2157382b-e7ba-44bf-9814-124cac52c539-maas-api-openshift-ca" pod "2157382b-e7ba-44bf-9814-124cac52c539" (UID: "2157382b-e7ba-44bf-9814-124cac52c539"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:48.099941 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:48.099921 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2157382b-e7ba-44bf-9814-124cac52c539-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:48.101395 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:48.101372 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2157382b-e7ba-44bf-9814-124cac52c539-kube-api-access-rglb2" pod "2157382b-e7ba-44bf-9814-124cac52c539" (UID: "2157382b-e7ba-44bf-9814-124cac52c539"). InnerVolumeSpecName "kube-api-access-rglb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:48.101475 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:48.101435 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2157382b-e7ba-44bf-9814-124cac52c539-maas-api-xks-ca" pod "2157382b-e7ba-44bf-9814-124cac52c539" (UID: "2157382b-e7ba-44bf-9814-124cac52c539"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:48.201868 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:48.201833 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2157382b-e7ba-44bf-9814-124cac52c539-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:48.201868 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:48.201864 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rglb2\" (UniqueName: \"kubernetes.io/projected/2157382b-e7ba-44bf-9814-124cac52c539-kube-api-access-rglb2\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:48.878186 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:48.878100 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" Nov 03 21:47:48.878186 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:48.878128 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n" event={"ID":"2157382b-e7ba-44bf-9814-124cac52c539","Type":"ContainerDied","Data":"b608d2dbf504005d6d97f3899ebd99a8992eaa232a11529a07f7f1ebedbd5eae"} Nov 03 21:47:48.878372 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:48.878220 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b608d2dbf504005d6d97f3899ebd99a8992eaa232a11529a07f7f1ebedbd5eae" Nov 03 21:47:48.879922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:48.879897 5041 generic.go:356] "Generic (PLEG): container finished" podID="5a88155b-e3ff-4308-94ee-d8cea2d3ed6a" containerID="af415b5d0ea5e8a7bccc79960ba63611e16ab419f510044ed4d041ecfdde14b9" exitCode=0 Nov 03 21:47:48.880031 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:48.879986 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt" event={"ID":"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a","Type":"ContainerDied","Data":"af415b5d0ea5e8a7bccc79960ba63611e16ab419f510044ed4d041ecfdde14b9"} Nov 03 21:47:49.071514 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.071489 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9" Nov 03 21:47:49.128047 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.116446 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-kube-api-access-n5bw7\" (UniqueName: \"kubernetes.io/projected/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-kube-api-access-n5bw7\") pod \"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651\" (UID: \"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651\") " Nov 03 21:47:49.128047 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.119828 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-kube-api-access-n5bw7" pod "a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651" (UID: "a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651"). InnerVolumeSpecName "kube-api-access-n5bw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:49.128047 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.125709 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-maas-api-xks-ca\") pod \"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651\" (UID: \"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651\") " Nov 03 21:47:49.137462 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.137366 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-maas-api-xks-ca" pod "a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651" (UID: "a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:49.228213 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.228170 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-maas-api-openshift-ca\") pod \"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651\" (UID: \"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651\") " Nov 03 21:47:49.228843 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.228815 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-maas-api-openshift-ca" pod "a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651" (UID: "a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:49.229022 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.228906 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5bw7\" (UniqueName: \"kubernetes.io/projected/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-kube-api-access-n5bw7\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:49.229133 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.229031 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:49.273302 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.273271 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm"] Nov 03 21:47:49.274091 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.274069 5041 cpu_manager.go:414] "containerMap: removing container" podUID="2157382b-e7ba-44bf-9814-124cac52c539" containerName="revoke-keys" Nov 03 21:47:49.274091 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.274091 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2157382b-e7ba-44bf-9814-124cac52c539" containerName="revoke-keys" Nov 03 21:47:49.274282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.274152 5041 cpu_manager.go:414] "containerMap: removing container" podUID="a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651" containerName="revoke-keys" Nov 03 21:47:49.274282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.274161 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651" containerName="revoke-keys" Nov 03 21:47:49.274395 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.274374 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="2157382b-e7ba-44bf-9814-124cac52c539" containerName="revoke-keys" Nov 03 21:47:49.274449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.274399 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651" containerName="revoke-keys" Nov 03 21:47:49.278507 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.278482 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" Nov 03 21:47:49.299369 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.298725 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm"] Nov 03 21:47:49.330690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.330634 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7e56bf-ba08-441e-925f-918461b36905-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm\" (UID: \"4d7e56bf-ba08-441e-925f-918461b36905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" Nov 03 21:47:49.330924 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.330834 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfn29\" (UniqueName: \"kubernetes.io/projected/4d7e56bf-ba08-441e-925f-918461b36905-kube-api-access-sfn29\") pod \"maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm\" (UID: \"4d7e56bf-ba08-441e-925f-918461b36905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" Nov 03 21:47:49.331038 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.331006 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7e56bf-ba08-441e-925f-918461b36905-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm\" (UID: \"4d7e56bf-ba08-441e-925f-918461b36905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" Nov 03 21:47:49.331499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.331161 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:49.436410 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.436286 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfn29\" (UniqueName: \"kubernetes.io/projected/4d7e56bf-ba08-441e-925f-918461b36905-kube-api-access-sfn29\") pod \"maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm\" (UID: \"4d7e56bf-ba08-441e-925f-918461b36905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" Nov 03 21:47:49.436595 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.436533 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7e56bf-ba08-441e-925f-918461b36905-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm\" (UID: \"4d7e56bf-ba08-441e-925f-918461b36905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" Nov 03 21:47:49.436749 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.436731 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7e56bf-ba08-441e-925f-918461b36905-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm\" (UID: \"4d7e56bf-ba08-441e-925f-918461b36905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" Nov 03 21:47:49.439729 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.437642 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7e56bf-ba08-441e-925f-918461b36905-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm\" (UID: \"4d7e56bf-ba08-441e-925f-918461b36905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" Nov 03 21:47:49.439729 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.439592 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7e56bf-ba08-441e-925f-918461b36905-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm\" (UID: \"4d7e56bf-ba08-441e-925f-918461b36905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" Nov 03 21:47:49.460506 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.460475 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfn29\" (UniqueName: \"kubernetes.io/projected/4d7e56bf-ba08-441e-925f-918461b36905-kube-api-access-sfn29\") pod \"maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm\" (UID: \"4d7e56bf-ba08-441e-925f-918461b36905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" Nov 03 21:47:49.609686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.607567 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" Nov 03 21:47:49.904686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:49.899745 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" podUID="47dc1e9a-eadd-46d0-8d2f-105d8fd8baba" containerName="payload-processing" containerID="cri-o://b9c0d27439af14c0779fcb4ad70bdb90f3285d4ad546bc7b121284090e01a1fd" gracePeriod=25 Nov 03 21:47:50.146700 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.144086 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9" Nov 03 21:47:50.148284 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.147239 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9" event={"ID":"a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651","Type":"ContainerDied","Data":"eb3e99dfefad948d7a3550800cb059e137e46b8f666f5d7cd5fccc581908f01a"} Nov 03 21:47:50.148284 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.147299 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb3e99dfefad948d7a3550800cb059e137e46b8f666f5d7cd5fccc581908f01a" Nov 03 21:47:50.189719 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:47:50.187218 5041 logging.go:55] [core] [Channel #677 SubChannel #678]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.71:50052", ServerName: "10.132.0.71:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.71:50052: connect: connection refused" Nov 03 21:47:50.255338 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:47:50.255301 5041 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fd0876_cfb0_4dd0_9e7f_0c3e49bfe651.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47dc1e9a_eadd_46d0_8d2f_105d8fd8baba.slice/crio-conmon-b9c0d27439af14c0779fcb4ad70bdb90f3285d4ad546bc7b121284090e01a1fd.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:47:50.342902 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.341884 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm"] Nov 03 21:47:50.448956 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.448567 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt" Nov 03 21:47:50.528060 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.527928 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" Nov 03 21:47:50.539786 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.539753 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-kube-api-access-g8j4v\" (UniqueName: \"kubernetes.io/projected/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-kube-api-access-g8j4v\") pod \"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a\" (UID: \"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a\") " Nov 03 21:47:50.540026 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.540008 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-maas-api-openshift-ca\") pod \"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a\" (UID: \"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a\") " Nov 03 21:47:50.540091 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.540075 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-maas-api-xks-ca\") pod \"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a\" (UID: \"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a\") " Nov 03 21:47:50.541571 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.541539 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-maas-api-openshift-ca" pod "5a88155b-e3ff-4308-94ee-d8cea2d3ed6a" (UID: "5a88155b-e3ff-4308-94ee-d8cea2d3ed6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:50.545048 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.545002 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-maas-api-xks-ca" pod "5a88155b-e3ff-4308-94ee-d8cea2d3ed6a" (UID: "5a88155b-e3ff-4308-94ee-d8cea2d3ed6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:50.548211 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.548183 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-kube-api-access-g8j4v" pod "5a88155b-e3ff-4308-94ee-d8cea2d3ed6a" (UID: "5a88155b-e3ff-4308-94ee-d8cea2d3ed6a"). InnerVolumeSpecName "kube-api-access-g8j4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:50.604361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.604323 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx"] Nov 03 21:47:50.679744 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.679712 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd5620-jbxvx"] Nov 03 21:47:50.707650 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.691609 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47dc1e9a-eadd-46d0-8d2f-105d8fd8baba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47dc1e9a-eadd-46d0-8d2f-105d8fd8baba-plugins-config-volume\") pod \"47dc1e9a-eadd-46d0-8d2f-105d8fd8baba\" (UID: \"47dc1e9a-eadd-46d0-8d2f-105d8fd8baba\") " Nov 03 21:47:50.718957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.718932 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47dc1e9a-eadd-46d0-8d2f-105d8fd8baba-kube-api-access-xbsjd\" (UniqueName: \"kubernetes.io/projected/47dc1e9a-eadd-46d0-8d2f-105d8fd8baba-kube-api-access-xbsjd\") pod \"47dc1e9a-eadd-46d0-8d2f-105d8fd8baba\" (UID: \"47dc1e9a-eadd-46d0-8d2f-105d8fd8baba\") " Nov 03 21:47:50.724737 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.695109 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47dc1e9a-eadd-46d0-8d2f-105d8fd8baba-plugins-config-volume" pod "47dc1e9a-eadd-46d0-8d2f-105d8fd8baba" (UID: "47dc1e9a-eadd-46d0-8d2f-105d8fd8baba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:50.724737 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.724305 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:50.724737 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.724332 5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47dc1e9a-eadd-46d0-8d2f-105d8fd8baba-plugins-config-volume\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:50.724737 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.724349 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8j4v\" (UniqueName: \"kubernetes.io/projected/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-kube-api-access-g8j4v\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:50.724737 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.724366 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:50.729926 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.729892 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47dc1e9a-eadd-46d0-8d2f-105d8fd8baba-kube-api-access-xbsjd" pod "47dc1e9a-eadd-46d0-8d2f-105d8fd8baba" (UID: "47dc1e9a-eadd-46d0-8d2f-105d8fd8baba"). InnerVolumeSpecName "kube-api-access-xbsjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:50.827050 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:50.827011 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbsjd\" (UniqueName: \"kubernetes.io/projected/47dc1e9a-eadd-46d0-8d2f-105d8fd8baba-kube-api-access-xbsjd\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:51.181797 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:51.181745 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" event={"ID":"4d7e56bf-ba08-441e-925f-918461b36905","Type":"ContainerStarted","Data":"e8257daa8319f30855856d96d1aa014d1caea78bb4e7446f109516c46daa93df"} Nov 03 21:47:51.182358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:51.181876 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" event={"ID":"4d7e56bf-ba08-441e-925f-918461b36905","Type":"ContainerStarted","Data":"d7283e128e74e8b89c820fb1b32f12b20c24c6c59c7d58aadfa8afa5881f5321"} Nov 03 21:47:51.184567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:51.184166 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" podUID="47dc1e9a-eadd-46d0-8d2f-105d8fd8baba" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.71:50052\" within 1s: context deadline exceeded" Nov 03 21:47:51.184744 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:51.184698 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt" event={"ID":"5a88155b-e3ff-4308-94ee-d8cea2d3ed6a","Type":"ContainerDied","Data":"b5bede310696b9e4e897fb33ddbd23119e545d63ebd080d312bf77d6e34f2fb7"} Nov 03 21:47:51.184798 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:51.184752 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5bede310696b9e4e897fb33ddbd23119e545d63ebd080d312bf77d6e34f2fb7" Nov 03 21:47:51.184843 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:51.184821 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt" Nov 03 21:47:51.187399 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:51.187067 5041 generic.go:356] "Generic (PLEG): container finished" podID="47dc1e9a-eadd-46d0-8d2f-105d8fd8baba" containerID="b9c0d27439af14c0779fcb4ad70bdb90f3285d4ad546bc7b121284090e01a1fd" exitCode=0 Nov 03 21:47:51.187399 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:51.187171 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" event={"ID":"47dc1e9a-eadd-46d0-8d2f-105d8fd8baba","Type":"ContainerDied","Data":"b9c0d27439af14c0779fcb4ad70bdb90f3285d4ad546bc7b121284090e01a1fd"} Nov 03 21:47:51.187399 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:51.187214 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" event={"ID":"47dc1e9a-eadd-46d0-8d2f-105d8fd8baba","Type":"ContainerDied","Data":"144971c6a0d0d37c02eab9b2a36efc28846e849a7020b9c5200a2cf387ebffb3"} Nov 03 21:47:51.187399 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:51.187244 5041 scope.go:122] "RemoveContainer" containerID="b9c0d27439af14c0779fcb4ad70bdb90f3285d4ad546bc7b121284090e01a1fd" Nov 03 21:47:51.187704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:51.187420 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9" Nov 03 21:47:51.202174 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:51.202155 5041 scope.go:122] "RemoveContainer" containerID="b9c0d27439af14c0779fcb4ad70bdb90f3285d4ad546bc7b121284090e01a1fd" Nov 03 21:47:51.202517 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:47:51.202488 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9c0d27439af14c0779fcb4ad70bdb90f3285d4ad546bc7b121284090e01a1fd\": container with ID starting with b9c0d27439af14c0779fcb4ad70bdb90f3285d4ad546bc7b121284090e01a1fd not found: ID does not exist" containerID="b9c0d27439af14c0779fcb4ad70bdb90f3285d4ad546bc7b121284090e01a1fd" Nov 03 21:47:51.202618 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:51.202522 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9c0d27439af14c0779fcb4ad70bdb90f3285d4ad546bc7b121284090e01a1fd"} err="failed to get container status \"b9c0d27439af14c0779fcb4ad70bdb90f3285d4ad546bc7b121284090e01a1fd\": rpc error: code = NotFound desc = could not find container \"b9c0d27439af14c0779fcb4ad70bdb90f3285d4ad546bc7b121284090e01a1fd\": container with ID starting with b9c0d27439af14c0779fcb4ad70bdb90f3285d4ad546bc7b121284090e01a1fd not found: ID does not exist" Nov 03 21:47:51.226501 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:51.226454 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" podStartSLOduration=2.226440256 podStartE2EDuration="2.226440256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:51.224843765 +0000 UTC m=+1086.732072060" watchObservedRunningTime="2026-11-03 21:47:51.226440256 +0000 UTC m=+1086.733668536" Nov 03 21:47:51.286006 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:51.285969 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9"] Nov 03 21:47:51.292979 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:51.292950 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e246f9-668fb6c698-bszl9"] Nov 03 21:47:52.198395 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:52.198356 5041 generic.go:356] "Generic (PLEG): container finished" podID="4d7e56bf-ba08-441e-925f-918461b36905" containerID="e8257daa8319f30855856d96d1aa014d1caea78bb4e7446f109516c46daa93df" exitCode=0 Nov 03 21:47:52.198840 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:52.198445 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" event={"ID":"4d7e56bf-ba08-441e-925f-918461b36905","Type":"ContainerDied","Data":"e8257daa8319f30855856d96d1aa014d1caea78bb4e7446f109516c46daa93df"} Nov 03 21:47:52.625411 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:52.625374 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47dc1e9a-eadd-46d0-8d2f-105d8fd8baba" path="/var/lib/kubelet/pods/47dc1e9a-eadd-46d0-8d2f-105d8fd8baba/volumes" Nov 03 21:47:52.626108 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:52.626082 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9266c8c-0d14-41f6-93ac-aecddca1ff89" path="/var/lib/kubelet/pods/d9266c8c-0d14-41f6-93ac-aecddca1ff89/volumes" Nov 03 21:47:53.392922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:53.392892 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" Nov 03 21:47:53.473360 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:53.473271 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d7e56bf-ba08-441e-925f-918461b36905-kube-api-access-sfn29\" (UniqueName: \"kubernetes.io/projected/4d7e56bf-ba08-441e-925f-918461b36905-kube-api-access-sfn29\") pod \"4d7e56bf-ba08-441e-925f-918461b36905\" (UID: \"4d7e56bf-ba08-441e-925f-918461b36905\") " Nov 03 21:47:53.473360 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:53.473335 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d7e56bf-ba08-441e-925f-918461b36905-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7e56bf-ba08-441e-925f-918461b36905-maas-api-openshift-ca\") pod \"4d7e56bf-ba08-441e-925f-918461b36905\" (UID: \"4d7e56bf-ba08-441e-925f-918461b36905\") " Nov 03 21:47:53.473580 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:53.473410 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d7e56bf-ba08-441e-925f-918461b36905-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7e56bf-ba08-441e-925f-918461b36905-maas-api-xks-ca\") pod \"4d7e56bf-ba08-441e-925f-918461b36905\" (UID: \"4d7e56bf-ba08-441e-925f-918461b36905\") " Nov 03 21:47:53.473999 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:53.473958 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d7e56bf-ba08-441e-925f-918461b36905-maas-api-openshift-ca" pod "4d7e56bf-ba08-441e-925f-918461b36905" (UID: "4d7e56bf-ba08-441e-925f-918461b36905"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:47:53.474621 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:53.474595 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7e56bf-ba08-441e-925f-918461b36905-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:53.475984 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:53.475960 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7e56bf-ba08-441e-925f-918461b36905-kube-api-access-sfn29" pod "4d7e56bf-ba08-441e-925f-918461b36905" (UID: "4d7e56bf-ba08-441e-925f-918461b36905"). InnerVolumeSpecName "kube-api-access-sfn29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:53.476545 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:53.476516 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d7e56bf-ba08-441e-925f-918461b36905-maas-api-xks-ca" pod "4d7e56bf-ba08-441e-925f-918461b36905" (UID: "4d7e56bf-ba08-441e-925f-918461b36905"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:53.576446 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:53.576411 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfn29\" (UniqueName: \"kubernetes.io/projected/4d7e56bf-ba08-441e-925f-918461b36905-kube-api-access-sfn29\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:53.576446 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:53.576443 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7e56bf-ba08-441e-925f-918461b36905-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:53.940962 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:53.940877 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf"] Nov 03 21:47:54.142604 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.142564 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" containerID="cri-o://2dc9f0723d698dfd82aa2d67882b0c0000ca3187d3d9b2208bb77a9cc7c80732" gracePeriod=45 Nov 03 21:47:54.224757 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.224717 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" event={"ID":"4d7e56bf-ba08-441e-925f-918461b36905","Type":"ContainerDied","Data":"d7283e128e74e8b89c820fb1b32f12b20c24c6c59c7d58aadfa8afa5881f5321"} Nov 03 21:47:54.224757 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.224763 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm" Nov 03 21:47:54.225025 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.224780 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7283e128e74e8b89c820fb1b32f12b20c24c6c59c7d58aadfa8afa5881f5321" Nov 03 21:47:54.302182 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.302146 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8"] Nov 03 21:47:54.492068 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.492004 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:47:54.592552 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.592518 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-dshm\") pod \"c9448e92-6c58-4dfe-93e9-7dc304b97066\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " Nov 03 21:47:54.592920 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.592901 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-model-cache\") pod \"c9448e92-6c58-4dfe-93e9-7dc304b97066\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " Nov 03 21:47:54.593028 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.592963 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9448e92-6c58-4dfe-93e9-7dc304b97066-kube-api-access-9k6fz\" (UniqueName: \"kubernetes.io/projected/c9448e92-6c58-4dfe-93e9-7dc304b97066-kube-api-access-9k6fz\") pod \"c9448e92-6c58-4dfe-93e9-7dc304b97066\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " Nov 03 21:47:54.593086 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.593062 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-home\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-home\") pod \"c9448e92-6c58-4dfe-93e9-7dc304b97066\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " Nov 03 21:47:54.593140 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.593120 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9448e92-6c58-4dfe-93e9-7dc304b97066-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9448e92-6c58-4dfe-93e9-7dc304b97066-tls-certs\") pod \"c9448e92-6c58-4dfe-93e9-7dc304b97066\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " Nov 03 21:47:54.593208 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.593197 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-tmp-dir\") pod \"c9448e92-6c58-4dfe-93e9-7dc304b97066\" (UID: \"c9448e92-6c58-4dfe-93e9-7dc304b97066\") " Nov 03 21:47:54.593710 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.593634 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-model-cache" pod "c9448e92-6c58-4dfe-93e9-7dc304b97066" (UID: "c9448e92-6c58-4dfe-93e9-7dc304b97066"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:54.594055 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.594023 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-tmp-dir" pod "c9448e92-6c58-4dfe-93e9-7dc304b97066" (UID: "c9448e92-6c58-4dfe-93e9-7dc304b97066"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:54.594249 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.594222 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-home" pod "c9448e92-6c58-4dfe-93e9-7dc304b97066" (UID: "c9448e92-6c58-4dfe-93e9-7dc304b97066"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:54.594526 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.594510 5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-model-cache\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:54.594602 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.594594 5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-home\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:54.594694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.594684 5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-tmp-dir\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:54.595808 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.595782 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-dshm" pod "c9448e92-6c58-4dfe-93e9-7dc304b97066" (UID: "c9448e92-6c58-4dfe-93e9-7dc304b97066"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:47:54.596810 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.596783 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9448e92-6c58-4dfe-93e9-7dc304b97066-kube-api-access-9k6fz" pod "c9448e92-6c58-4dfe-93e9-7dc304b97066" (UID: "c9448e92-6c58-4dfe-93e9-7dc304b97066"). InnerVolumeSpecName "kube-api-access-9k6fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:47:54.598040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.598015 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9448e92-6c58-4dfe-93e9-7dc304b97066-tls-certs" pod "c9448e92-6c58-4dfe-93e9-7dc304b97066" (UID: "c9448e92-6c58-4dfe-93e9-7dc304b97066"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:47:54.696289 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.696256 5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9448e92-6c58-4dfe-93e9-7dc304b97066-tls-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:54.696289 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.696286 5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9448e92-6c58-4dfe-93e9-7dc304b97066-dshm\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:54.696289 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:54.696297 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k6fz\" (UniqueName: \"kubernetes.io/projected/c9448e92-6c58-4dfe-93e9-7dc304b97066-kube-api-access-9k6fz\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:47:55.238670 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:55.238628 5041 generic.go:356] "Generic (PLEG): container finished" podID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerID="2dc9f0723d698dfd82aa2d67882b0c0000ca3187d3d9b2208bb77a9cc7c80732" exitCode=0 Nov 03 21:47:55.238855 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:55.238716 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" event={"ID":"c9448e92-6c58-4dfe-93e9-7dc304b97066","Type":"ContainerDied","Data":"2dc9f0723d698dfd82aa2d67882b0c0000ca3187d3d9b2208bb77a9cc7c80732"} Nov 03 21:47:55.238855 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:55.238745 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" Nov 03 21:47:55.238855 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:55.238763 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" event={"ID":"c9448e92-6c58-4dfe-93e9-7dc304b97066","Type":"ContainerDied","Data":"1664fee5984ad0d413f91170b69f703579e1bf50a89304f162c788324f8dccb5"} Nov 03 21:47:55.238855 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:55.238783 5041 scope.go:122] "RemoveContainer" containerID="2dc9f0723d698dfd82aa2d67882b0c0000ca3187d3d9b2208bb77a9cc7c80732" Nov 03 21:47:55.252833 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:55.252807 5041 scope.go:122] "RemoveContainer" containerID="2dc9f0723d698dfd82aa2d67882b0c0000ca3187d3d9b2208bb77a9cc7c80732" Nov 03 21:47:55.253200 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:47:55.253176 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc9f0723d698dfd82aa2d67882b0c0000ca3187d3d9b2208bb77a9cc7c80732\": container with ID starting with 2dc9f0723d698dfd82aa2d67882b0c0000ca3187d3d9b2208bb77a9cc7c80732 not found: ID does not exist" containerID="2dc9f0723d698dfd82aa2d67882b0c0000ca3187d3d9b2208bb77a9cc7c80732" Nov 03 21:47:55.253330 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:55.253206 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc9f0723d698dfd82aa2d67882b0c0000ca3187d3d9b2208bb77a9cc7c80732"} err="failed to get container status \"2dc9f0723d698dfd82aa2d67882b0c0000ca3187d3d9b2208bb77a9cc7c80732\": rpc error: code = NotFound desc = could not find container \"2dc9f0723d698dfd82aa2d67882b0c0000ca3187d3d9b2208bb77a9cc7c80732\": container with ID starting with 2dc9f0723d698dfd82aa2d67882b0c0000ca3187d3d9b2208bb77a9cc7c80732 not found: ID does not exist" Nov 03 21:47:55.298149 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:55.298117 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4"] Nov 03 21:47:55.304683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:55.304636 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4"] Nov 03 21:47:55.440525 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:55.440458 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e246f9/distinct-2-w0-e246f9-kserve-64869f999d-qtdj4" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:56.546700 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.543761 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh"] Nov 03 21:47:56.601607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.597710 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz"] Nov 03 21:47:56.601607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.598870 5041 cpu_manager.go:414] "containerMap: removing container" podUID="5a88155b-e3ff-4308-94ee-d8cea2d3ed6a" containerName="revoke-keys" Nov 03 21:47:56.601607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.598903 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a88155b-e3ff-4308-94ee-d8cea2d3ed6a" containerName="revoke-keys" Nov 03 21:47:56.601607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.598954 5041 cpu_manager.go:414] "containerMap: removing container" podUID="47dc1e9a-eadd-46d0-8d2f-105d8fd8baba" containerName="payload-processing" Nov 03 21:47:56.601607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.598962 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47dc1e9a-eadd-46d0-8d2f-105d8fd8baba" containerName="payload-processing" Nov 03 21:47:56.601607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.598983 5041 cpu_manager.go:414] "containerMap: removing container" podUID="4d7e56bf-ba08-441e-925f-918461b36905" containerName="revoke-keys" Nov 03 21:47:56.601607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.598991 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d7e56bf-ba08-441e-925f-918461b36905" containerName="revoke-keys" Nov 03 21:47:56.601607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.599013 5041 cpu_manager.go:414] "containerMap: removing container" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" Nov 03 21:47:56.601607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.599020 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" Nov 03 21:47:56.601607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.599183 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d7e56bf-ba08-441e-925f-918461b36905" containerName="revoke-keys" Nov 03 21:47:56.601607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.599200 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" containerName="main" Nov 03 21:47:56.601607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.599212 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a88155b-e3ff-4308-94ee-d8cea2d3ed6a" containerName="revoke-keys" Nov 03 21:47:56.601607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.599225 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="47dc1e9a-eadd-46d0-8d2f-105d8fd8baba" containerName="payload-processing" Nov 03 21:47:56.672682 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.634838 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" Nov 03 21:47:56.710683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.673213 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:47:56.710683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.673932 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:47:56.710683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.694133 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9448e92-6c58-4dfe-93e9-7dc304b97066" path="/var/lib/kubelet/pods/c9448e92-6c58-4dfe-93e9-7dc304b97066/volumes" Nov 03 21:47:56.973111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.969496 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz"] Nov 03 21:47:56.973111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.972190 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj7ms\" (UniqueName: \"kubernetes.io/projected/76d6f982-faed-4a04-aa25-85a2159c343b-kube-api-access-bj7ms\") pod \"maas-api-revoke-sub-b5290ae2-zmmpz\" (UID: \"76d6f982-faed-4a04-aa25-85a2159c343b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" Nov 03 21:47:56.981687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.977826 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76d6f982-faed-4a04-aa25-85a2159c343b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5290ae2-zmmpz\" (UID: \"76d6f982-faed-4a04-aa25-85a2159c343b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" Nov 03 21:47:56.981687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:56.977942 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76d6f982-faed-4a04-aa25-85a2159c343b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5290ae2-zmmpz\" (UID: \"76d6f982-faed-4a04-aa25-85a2159c343b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" Nov 03 21:47:57.095689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.089204 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76d6f982-faed-4a04-aa25-85a2159c343b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5290ae2-zmmpz\" (UID: \"76d6f982-faed-4a04-aa25-85a2159c343b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" Nov 03 21:47:57.095689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.089296 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76d6f982-faed-4a04-aa25-85a2159c343b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5290ae2-zmmpz\" (UID: \"76d6f982-faed-4a04-aa25-85a2159c343b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" Nov 03 21:47:57.095689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.089653 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj7ms\" (UniqueName: \"kubernetes.io/projected/76d6f982-faed-4a04-aa25-85a2159c343b-kube-api-access-bj7ms\") pod \"maas-api-revoke-sub-b5290ae2-zmmpz\" (UID: \"76d6f982-faed-4a04-aa25-85a2159c343b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" Nov 03 21:47:57.297773 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.142886 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76d6f982-faed-4a04-aa25-85a2159c343b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5290ae2-zmmpz\" (UID: \"76d6f982-faed-4a04-aa25-85a2159c343b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" Nov 03 21:47:57.297773 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.190407 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6"] Nov 03 21:47:57.335492 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.335458 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" Nov 03 21:47:57.362951 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.362917 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76d6f982-faed-4a04-aa25-85a2159c343b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5290ae2-zmmpz\" (UID: \"76d6f982-faed-4a04-aa25-85a2159c343b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" Nov 03 21:47:57.394469 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.394387 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj7ms\" (UniqueName: \"kubernetes.io/projected/76d6f982-faed-4a04-aa25-85a2159c343b-kube-api-access-bj7ms\") pod \"maas-api-revoke-sub-b5290ae2-zmmpz\" (UID: \"76d6f982-faed-4a04-aa25-85a2159c343b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" Nov 03 21:47:57.411765 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.411725 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr6v4\" (UniqueName: \"kubernetes.io/projected/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-kube-api-access-xr6v4\") pod \"maas-api-revoke-sub-6587b599-x8xn6\" (UID: \"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" Nov 03 21:47:57.412104 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.412078 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6587b599-x8xn6\" (UID: \"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" Nov 03 21:47:57.412477 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.412448 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6587b599-x8xn6\" (UID: \"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" Nov 03 21:47:57.517696 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.513608 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr6v4\" (UniqueName: \"kubernetes.io/projected/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-kube-api-access-xr6v4\") pod \"maas-api-revoke-sub-6587b599-x8xn6\" (UID: \"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" Nov 03 21:47:57.538701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.538189 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6587b599-x8xn6\" (UID: \"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" Nov 03 21:47:57.538701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.538530 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6587b599-x8xn6\" (UID: \"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" Nov 03 21:47:57.540702 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.539514 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6587b599-x8xn6\" (UID: \"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" Nov 03 21:47:57.573400 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.573323 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6587b599-x8xn6\" (UID: \"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" Nov 03 21:47:57.603904 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.603869 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr6v4\" (UniqueName: \"kubernetes.io/projected/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-kube-api-access-xr6v4\") pod \"maas-api-revoke-sub-6587b599-x8xn6\" (UID: \"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" Nov 03 21:47:57.613219 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.612940 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" Nov 03 21:47:57.743438 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:57.743403 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6"] Nov 03 21:47:58.032140 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:58.032111 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" Nov 03 21:47:58.858398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:58.857760 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 21:47:59.169775 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.158219 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:47:59.169775 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.158305 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:59.169775 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.159138 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:59.322703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.300942 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:59.351684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.325373 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" podUID="f05ce451-3614-4af6-88e8-8793cf631ea1" containerName="payload-processing" containerID="cri-o://777cab1a0a4603d3b57dbc3d194b833a766e06e25e54dcefefd054ae152d1276" gracePeriod=25 Nov 03 21:47:59.668956 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:47:59.658483 5041 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.025s" Nov 03 21:47:59.668956 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.658540 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d"] Nov 03 21:47:59.734794 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.716810 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.120:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:47:59.793498 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.788974 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" Nov 03 21:47:59.824867 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.804599 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f38640-6dc3-47b9-8e3b-36efdbd57c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bdf0ab2-9zr4d\" (UID: \"56f38640-6dc3-47b9-8e3b-36efdbd57c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" Nov 03 21:47:59.826552 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.826518 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dzvc\" (UniqueName: \"kubernetes.io/projected/56f38640-6dc3-47b9-8e3b-36efdbd57c68-kube-api-access-9dzvc\") pod \"maas-api-revoke-sub-1bdf0ab2-9zr4d\" (UID: \"56f38640-6dc3-47b9-8e3b-36efdbd57c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" Nov 03 21:47:59.827195 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.827171 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f38640-6dc3-47b9-8e3b-36efdbd57c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bdf0ab2-9zr4d\" (UID: \"56f38640-6dc3-47b9-8e3b-36efdbd57c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" Nov 03 21:47:59.875689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.861768 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 21:47:59.960220 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.899368 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:47:59.963701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.962581 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f38640-6dc3-47b9-8e3b-36efdbd57c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bdf0ab2-9zr4d\" (UID: \"56f38640-6dc3-47b9-8e3b-36efdbd57c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" Nov 03 21:47:59.963701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.962644 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dzvc\" (UniqueName: \"kubernetes.io/projected/56f38640-6dc3-47b9-8e3b-36efdbd57c68-kube-api-access-9dzvc\") pod \"maas-api-revoke-sub-1bdf0ab2-9zr4d\" (UID: \"56f38640-6dc3-47b9-8e3b-36efdbd57c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" Nov 03 21:47:59.963701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.963139 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f38640-6dc3-47b9-8e3b-36efdbd57c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bdf0ab2-9zr4d\" (UID: \"56f38640-6dc3-47b9-8e3b-36efdbd57c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" Nov 03 21:47:59.964693 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:47:59.964621 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f38640-6dc3-47b9-8e3b-36efdbd57c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bdf0ab2-9zr4d\" (UID: \"56f38640-6dc3-47b9-8e3b-36efdbd57c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" Nov 03 21:48:00.199014 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:00.159288 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d"] Nov 03 21:48:00.260818 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:00.211361 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f38640-6dc3-47b9-8e3b-36efdbd57c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bdf0ab2-9zr4d\" (UID: \"56f38640-6dc3-47b9-8e3b-36efdbd57c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" Nov 03 21:48:00.260818 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:00.215532 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 21:48:00.344371 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:00.262167 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:48:00.373472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:00.373430 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Nov 03 21:48:00.462105 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:00.441098 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n"] Nov 03 21:48:00.462560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:00.462529 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" podUID="5ba2a45a-0811-49ae-84db-468f7950fbb6" containerName="istio-proxy" containerID="cri-o://1a23f6899f664aa83630f2f57214edf337bfe9c30801ab7695962debf8dd89fe" gracePeriod=30 Nov 03 21:48:00.462879 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:00.441162 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:48:00.529917 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:00.529814 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" podUID="f05ce451-3614-4af6-88e8-8793cf631ea1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:48:00.774844 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:00.717517 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dzvc\" (UniqueName: \"kubernetes.io/projected/56f38640-6dc3-47b9-8e3b-36efdbd57c68-kube-api-access-9dzvc\") pod \"maas-api-revoke-sub-1bdf0ab2-9zr4d\" (UID: \"56f38640-6dc3-47b9-8e3b-36efdbd57c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" Nov 03 21:48:00.918831 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:00.775135 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.120:8000/ready\": context deadline exceeded" Nov 03 21:48:00.918831 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:00.907910 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:48:01.085688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:01.085310 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" Nov 03 21:48:01.243509 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:01.228769 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h"] Nov 03 21:48:01.243509 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:01.228877 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:48:01.243509 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:01.228898 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd"] Nov 03 21:48:01.243509 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:01.229105 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" podUID="d6132e6c-086e-47bc-9baa-4a341f85671c" containerName="istio-proxy" containerID="cri-o://320a8f899cc06d57d818f2b5f37146911aff2531e13003dcee0188e63603562e" gracePeriod=30 Nov 03 21:48:01.243509 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:01.229750 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" podUID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" containerName="istio-proxy" containerID="cri-o://294a95b54fb0ae20b17cbeaad31f1d59674f2a7b21ba002703a77e6c206136a5" gracePeriod=30 Nov 03 21:48:01.321004 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:01.273806 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 21:48:01.406841 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:01.406789 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:48:01.631834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:01.629819 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" podUID="cabaa235-b28f-41a4-ad06-a10e60adc05e" containerName="payload-processing" containerID="cri-o://8702e5d74d46857bffc7c087b017643103bdd1e81d44ab751757d55c43f01d41" gracePeriod=25 Nov 03 21:48:01.814370 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:01.788777 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:48:01.835947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:01.835905 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:48:01.863749 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:01.859767 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.120:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:48:01.939744 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:01.929139 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:48:02.535107 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:02.285377 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 21:48:02.535107 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:02.451471 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 21:48:02.629411 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:02.560921 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:48:02.763686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:02.751709 5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": context deadline exceeded" Nov 03 21:48:02.804626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:02.803414 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:48:03.368315 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:03.344239 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.120:8000/ready\": context deadline exceeded" Nov 03 21:48:03.368315 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:03.364981 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-v2pwb" podUID="fa892a3f-4760-45ff-b00b-0840970540bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 21:48:03.462223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:03.462169 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:48:03.619739 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:03.601996 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 21:48:03.709391 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:03.706229 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Nov 03 21:48:03.833463 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:03.833422 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:48:04.010540 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:48:04.010434 5041 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.334s" Nov 03 21:48:04.050785 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:04.015841 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:48:04.050785 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:04.015911 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:48:04.050785 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:04.015968 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:48:04.074421 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:04.074066 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:48:04.084850 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:04.084818 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" event={"ID":"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93","Type":"ContainerStarted","Data":"83f7034467c4e9b6336ca30b42e8c31cc506097acc80c0ec91a546ccfd881d46"} Nov 03 21:48:04.092913 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:04.092868 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" event={"ID":"76d6f982-faed-4a04-aa25-85a2159c343b","Type":"ContainerStarted","Data":"8643a998aa2f9f644c3633257c8506736c500ce0acbf7378a29ee6c89863c6ff"} Nov 03 21:48:04.106905 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:04.098023 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:48:04.123528 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:04.123473 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz"] Nov 03 21:48:04.136877 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:04.136747 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6"] Nov 03 21:48:04.326156 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:04.326121 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d"] Nov 03 21:48:05.106767 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:05.106656 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" event={"ID":"56f38640-6dc3-47b9-8e3b-36efdbd57c68","Type":"ContainerStarted","Data":"9da7a13156bff0c32521416b328df37c79ce771b02d6d0419c34ddc85d1a8860"} Nov 03 21:48:05.106767 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:05.106739 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" event={"ID":"56f38640-6dc3-47b9-8e3b-36efdbd57c68","Type":"ContainerStarted","Data":"3ada30354791e619d46eee79014d3f176c60b4202df0da9676343e56993de94b"} Nov 03 21:48:05.119732 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:05.111543 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" event={"ID":"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93","Type":"ContainerStarted","Data":"431a999d092bf3a8241e74ea0d26e7821eaccae7f3fdb424fa08bf2488724704"} Nov 03 21:48:05.119732 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:05.114695 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" event={"ID":"76d6f982-faed-4a04-aa25-85a2159c343b","Type":"ContainerStarted","Data":"51fd43d50e100e6c571479549e5ff91985c07f4a2af0e62a18e9daac92dd309c"} Nov 03 21:48:05.151263 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:05.151215 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" podStartSLOduration=8.151197269 podStartE2EDuration="8.151197269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:05.148172119 +0000 UTC m=+1100.655400399" watchObservedRunningTime="2026-11-03 21:48:05.151197269 +0000 UTC m=+1100.658425583" Nov 03 21:48:05.181923 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:05.181885 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:48:05.182083 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:05.181950 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" podUID="d6132e6c-086e-47bc-9baa-4a341f85671c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:48:05.189515 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:05.189465 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" podStartSLOduration=9.189451019 podStartE2EDuration="9.189451019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:05.188850025 +0000 UTC m=+1100.696078318" watchObservedRunningTime="2026-11-03 21:48:05.189451019 +0000 UTC m=+1100.696679299" Nov 03 21:48:05.233316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:05.233260 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" podStartSLOduration=9.233244453 podStartE2EDuration="9.233244453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:05.229270681 +0000 UTC m=+1100.736498961" watchObservedRunningTime="2026-11-03 21:48:05.233244453 +0000 UTC m=+1100.740472727" Nov 03 21:48:06.159245 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:06.159193 5041 generic.go:356] "Generic (PLEG): container finished" podID="d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93" containerID="431a999d092bf3a8241e74ea0d26e7821eaccae7f3fdb424fa08bf2488724704" exitCode=0 Nov 03 21:48:06.159822 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:06.159403 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" event={"ID":"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93","Type":"ContainerDied","Data":"431a999d092bf3a8241e74ea0d26e7821eaccae7f3fdb424fa08bf2488724704"} Nov 03 21:48:06.176679 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:06.176631 5041 generic.go:356] "Generic (PLEG): container finished" podID="76d6f982-faed-4a04-aa25-85a2159c343b" containerID="51fd43d50e100e6c571479549e5ff91985c07f4a2af0e62a18e9daac92dd309c" exitCode=0 Nov 03 21:48:06.177011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:06.176966 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" event={"ID":"76d6f982-faed-4a04-aa25-85a2159c343b","Type":"ContainerDied","Data":"51fd43d50e100e6c571479549e5ff91985c07f4a2af0e62a18e9daac92dd309c"} Nov 03 21:48:06.178921 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:06.178897 5041 generic.go:356] "Generic (PLEG): container finished" podID="56f38640-6dc3-47b9-8e3b-36efdbd57c68" containerID="9da7a13156bff0c32521416b328df37c79ce771b02d6d0419c34ddc85d1a8860" exitCode=0 Nov 03 21:48:06.179042 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:06.178954 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" event={"ID":"56f38640-6dc3-47b9-8e3b-36efdbd57c68","Type":"ContainerDied","Data":"9da7a13156bff0c32521416b328df37c79ce771b02d6d0419c34ddc85d1a8860"} Nov 03 21:48:06.261935 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:06.261897 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2"] Nov 03 21:48:06.269457 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:06.269419 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h"] Nov 03 21:48:06.351168 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:06.351128 5041 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:48:06.351360 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:06.351203 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" podUID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:48:07.026556 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.026498 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql"] Nov 03 21:48:07.034782 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.034736 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" Nov 03 21:48:07.058968 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.058937 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql"] Nov 03 21:48:07.061589 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.061546 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1561a7cd-gstql\" (UID: \"3a8a6d82-56a3-4ef0-9785-f9578579b5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" Nov 03 21:48:07.061843 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.061812 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vrkf\" (UniqueName: \"kubernetes.io/projected/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-kube-api-access-7vrkf\") pod \"maas-api-revoke-sub-1561a7cd-gstql\" (UID: \"3a8a6d82-56a3-4ef0-9785-f9578579b5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" Nov 03 21:48:07.061935 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.061863 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1561a7cd-gstql\" (UID: \"3a8a6d82-56a3-4ef0-9785-f9578579b5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" Nov 03 21:48:07.063196 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.063165 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" podUID="cabaa235-b28f-41a4-ad06-a10e60adc05e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:48:07.163292 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.163251 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1561a7cd-gstql\" (UID: \"3a8a6d82-56a3-4ef0-9785-f9578579b5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" Nov 03 21:48:07.163720 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.163369 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vrkf\" (UniqueName: \"kubernetes.io/projected/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-kube-api-access-7vrkf\") pod \"maas-api-revoke-sub-1561a7cd-gstql\" (UID: \"3a8a6d82-56a3-4ef0-9785-f9578579b5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" Nov 03 21:48:07.163720 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.163399 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1561a7cd-gstql\" (UID: \"3a8a6d82-56a3-4ef0-9785-f9578579b5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" Nov 03 21:48:07.164066 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.164044 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1561a7cd-gstql\" (UID: \"3a8a6d82-56a3-4ef0-9785-f9578579b5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" Nov 03 21:48:07.165867 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.165840 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1561a7cd-gstql\" (UID: \"3a8a6d82-56a3-4ef0-9785-f9578579b5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" Nov 03 21:48:07.184546 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.184517 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vrkf\" (UniqueName: \"kubernetes.io/projected/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-kube-api-access-7vrkf\") pod \"maas-api-revoke-sub-1561a7cd-gstql\" (UID: \"3a8a6d82-56a3-4ef0-9785-f9578579b5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" Nov 03 21:48:07.360484 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.360409 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" Nov 03 21:48:07.443648 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.442055 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" Nov 03 21:48:07.472429 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.468897 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-maas-api-xks-ca\") pod \"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93\" (UID: \"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93\") " Nov 03 21:48:07.472429 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.469084 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-maas-api-openshift-ca\") pod \"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93\" (UID: \"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93\") " Nov 03 21:48:07.472429 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.469177 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-kube-api-access-xr6v4\" (UniqueName: \"kubernetes.io/projected/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-kube-api-access-xr6v4\") pod \"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93\" (UID: \"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93\") " Nov 03 21:48:07.472429 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.470636 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-maas-api-openshift-ca" pod "d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93" (UID: "d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:07.492882 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.492848 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-maas-api-xks-ca" pod "d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93" (UID: "d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:07.506321 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.506280 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-kube-api-access-xr6v4" pod "d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93" (UID: "d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93"). InnerVolumeSpecName "kube-api-access-xr6v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:07.521752 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.514111 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" Nov 03 21:48:07.537193 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.537169 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" Nov 03 21:48:07.584867 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.579061 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:07.584867 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.579105 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr6v4\" (UniqueName: \"kubernetes.io/projected/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-kube-api-access-xr6v4\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:07.584867 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.579122 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:07.679949 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.679902 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f38640-6dc3-47b9-8e3b-36efdbd57c68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f38640-6dc3-47b9-8e3b-36efdbd57c68-maas-api-openshift-ca\") pod \"56f38640-6dc3-47b9-8e3b-36efdbd57c68\" (UID: \"56f38640-6dc3-47b9-8e3b-36efdbd57c68\") " Nov 03 21:48:07.680127 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.679987 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76d6f982-faed-4a04-aa25-85a2159c343b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76d6f982-faed-4a04-aa25-85a2159c343b-maas-api-xks-ca\") pod \"76d6f982-faed-4a04-aa25-85a2159c343b\" (UID: \"76d6f982-faed-4a04-aa25-85a2159c343b\") " Nov 03 21:48:07.680209 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.680142 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d6f982-faed-4a04-aa25-85a2159c343b-kube-api-access-bj7ms\" (UniqueName: \"kubernetes.io/projected/76d6f982-faed-4a04-aa25-85a2159c343b-kube-api-access-bj7ms\") pod \"76d6f982-faed-4a04-aa25-85a2159c343b\" (UID: \"76d6f982-faed-4a04-aa25-85a2159c343b\") " Nov 03 21:48:07.680354 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.680331 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f38640-6dc3-47b9-8e3b-36efdbd57c68-kube-api-access-9dzvc\" (UniqueName: \"kubernetes.io/projected/56f38640-6dc3-47b9-8e3b-36efdbd57c68-kube-api-access-9dzvc\") pod \"56f38640-6dc3-47b9-8e3b-36efdbd57c68\" (UID: \"56f38640-6dc3-47b9-8e3b-36efdbd57c68\") " Nov 03 21:48:07.680505 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.680443 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56f38640-6dc3-47b9-8e3b-36efdbd57c68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f38640-6dc3-47b9-8e3b-36efdbd57c68-maas-api-xks-ca\") pod \"56f38640-6dc3-47b9-8e3b-36efdbd57c68\" (UID: \"56f38640-6dc3-47b9-8e3b-36efdbd57c68\") " Nov 03 21:48:07.680599 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.680330 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f38640-6dc3-47b9-8e3b-36efdbd57c68-maas-api-openshift-ca" pod "56f38640-6dc3-47b9-8e3b-36efdbd57c68" (UID: "56f38640-6dc3-47b9-8e3b-36efdbd57c68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:07.680599 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.680538 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76d6f982-faed-4a04-aa25-85a2159c343b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76d6f982-faed-4a04-aa25-85a2159c343b-maas-api-openshift-ca\") pod \"76d6f982-faed-4a04-aa25-85a2159c343b\" (UID: \"76d6f982-faed-4a04-aa25-85a2159c343b\") " Nov 03 21:48:07.681047 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.681019 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d6f982-faed-4a04-aa25-85a2159c343b-maas-api-openshift-ca" pod "76d6f982-faed-4a04-aa25-85a2159c343b" (UID: "76d6f982-faed-4a04-aa25-85a2159c343b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:07.681309 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.681284 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76d6f982-faed-4a04-aa25-85a2159c343b-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:07.681423 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.681379 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56f38640-6dc3-47b9-8e3b-36efdbd57c68-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:07.682982 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.682958 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56f38640-6dc3-47b9-8e3b-36efdbd57c68-maas-api-xks-ca" pod "56f38640-6dc3-47b9-8e3b-36efdbd57c68" (UID: "56f38640-6dc3-47b9-8e3b-36efdbd57c68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:07.683092 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.683025 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d6f982-faed-4a04-aa25-85a2159c343b-kube-api-access-bj7ms" pod "76d6f982-faed-4a04-aa25-85a2159c343b" (UID: "76d6f982-faed-4a04-aa25-85a2159c343b"). InnerVolumeSpecName "kube-api-access-bj7ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:07.683613 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.683590 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76d6f982-faed-4a04-aa25-85a2159c343b-maas-api-xks-ca" pod "76d6f982-faed-4a04-aa25-85a2159c343b" (UID: "76d6f982-faed-4a04-aa25-85a2159c343b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:07.684132 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.684110 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f38640-6dc3-47b9-8e3b-36efdbd57c68-kube-api-access-9dzvc" pod "56f38640-6dc3-47b9-8e3b-36efdbd57c68" (UID: "56f38640-6dc3-47b9-8e3b-36efdbd57c68"). InnerVolumeSpecName "kube-api-access-9dzvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:07.739944 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:48:07.739904 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8a6d82_56a3_4ef0_9785_f9578579b5f0.slice/crio-5b5cd9e2f1ed25eb9ecf4f57118c72ab6dab27a4e0d88fd207a24e9055e6e6b0 WatchSource:0}: Error finding container 5b5cd9e2f1ed25eb9ecf4f57118c72ab6dab27a4e0d88fd207a24e9055e6e6b0: Status 404 returned error can't find the container with id 5b5cd9e2f1ed25eb9ecf4f57118c72ab6dab27a4e0d88fd207a24e9055e6e6b0 Nov 03 21:48:07.741761 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.741731 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql"] Nov 03 21:48:07.783496 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.783467 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76d6f982-faed-4a04-aa25-85a2159c343b-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:07.783496 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.783496 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj7ms\" (UniqueName: \"kubernetes.io/projected/76d6f982-faed-4a04-aa25-85a2159c343b-kube-api-access-bj7ms\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:07.783693 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.783506 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dzvc\" (UniqueName: \"kubernetes.io/projected/56f38640-6dc3-47b9-8e3b-36efdbd57c68-kube-api-access-9dzvc\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:07.783693 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:07.783515 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56f38640-6dc3-47b9-8e3b-36efdbd57c68-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.211964 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.211512 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:48:08.219234 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.216974 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" event={"ID":"d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93","Type":"ContainerDied","Data":"83f7034467c4e9b6336ca30b42e8c31cc506097acc80c0ec91a546ccfd881d46"} Nov 03 21:48:08.219234 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.217076 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83f7034467c4e9b6336ca30b42e8c31cc506097acc80c0ec91a546ccfd881d46" Nov 03 21:48:08.219234 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.217180 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6" Nov 03 21:48:08.230354 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.224721 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" event={"ID":"76d6f982-faed-4a04-aa25-85a2159c343b","Type":"ContainerDied","Data":"8643a998aa2f9f644c3633257c8506736c500ce0acbf7378a29ee6c89863c6ff"} Nov 03 21:48:08.230354 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.224795 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8643a998aa2f9f644c3633257c8506736c500ce0acbf7378a29ee6c89863c6ff" Nov 03 21:48:08.230354 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.224891 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz" Nov 03 21:48:08.241685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.240966 5041 generic.go:356] "Generic (PLEG): container finished" podID="d6132e6c-086e-47bc-9baa-4a341f85671c" containerID="320a8f899cc06d57d818f2b5f37146911aff2531e13003dcee0188e63603562e" exitCode=0 Nov 03 21:48:08.241685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.241122 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" event={"ID":"d6132e6c-086e-47bc-9baa-4a341f85671c","Type":"ContainerDied","Data":"320a8f899cc06d57d818f2b5f37146911aff2531e13003dcee0188e63603562e"} Nov 03 21:48:08.260436 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.245563 5041 generic.go:356] "Generic (PLEG): container finished" podID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" containerID="294a95b54fb0ae20b17cbeaad31f1d59674f2a7b21ba002703a77e6c206136a5" exitCode=0 Nov 03 21:48:08.260436 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.245705 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" event={"ID":"4dcc2293-9b7c-4381-92f2-1657ee9a1e29","Type":"ContainerDied","Data":"294a95b54fb0ae20b17cbeaad31f1d59674f2a7b21ba002703a77e6c206136a5"} Nov 03 21:48:08.260436 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.245760 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" event={"ID":"4dcc2293-9b7c-4381-92f2-1657ee9a1e29","Type":"ContainerDied","Data":"cea920656c09828b68820e8229412800bb1aa4b633675e0fc4b50ee072b22039"} Nov 03 21:48:08.260436 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.245793 5041 scope.go:122] "RemoveContainer" containerID="294a95b54fb0ae20b17cbeaad31f1d59674f2a7b21ba002703a77e6c206136a5" Nov 03 21:48:08.260436 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.245979 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h" Nov 03 21:48:08.260436 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.254819 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" event={"ID":"3a8a6d82-56a3-4ef0-9785-f9578579b5f0","Type":"ContainerStarted","Data":"d4acb8a9d294b980b519a5b62cb67f8ef3ee45e583e4746a15ce01a1914ad428"} Nov 03 21:48:08.260436 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.255060 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" event={"ID":"3a8a6d82-56a3-4ef0-9785-f9578579b5f0","Type":"ContainerStarted","Data":"5b5cd9e2f1ed25eb9ecf4f57118c72ab6dab27a4e0d88fd207a24e9055e6e6b0"} Nov 03 21:48:08.263011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.262976 5041 generic.go:356] "Generic (PLEG): container finished" podID="5ba2a45a-0811-49ae-84db-468f7950fbb6" containerID="1a23f6899f664aa83630f2f57214edf337bfe9c30801ab7695962debf8dd89fe" exitCode=0 Nov 03 21:48:08.263145 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.263084 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" event={"ID":"5ba2a45a-0811-49ae-84db-468f7950fbb6","Type":"ContainerDied","Data":"1a23f6899f664aa83630f2f57214edf337bfe9c30801ab7695962debf8dd89fe"} Nov 03 21:48:08.270528 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.267172 5041 scope.go:122] "RemoveContainer" containerID="24a652966f936844adcfac8963d00e9fb6751cc2993ee5a29dd0344e8236fa58" Nov 03 21:48:08.270528 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.267862 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" event={"ID":"56f38640-6dc3-47b9-8e3b-36efdbd57c68","Type":"ContainerDied","Data":"3ada30354791e619d46eee79014d3f176c60b4202df0da9676343e56993de94b"} Nov 03 21:48:08.270528 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.267911 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ada30354791e619d46eee79014d3f176c60b4202df0da9676343e56993de94b" Nov 03 21:48:08.270528 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.268161 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d" Nov 03 21:48:08.295211 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.295173 5041 scope.go:122] "RemoveContainer" containerID="294a95b54fb0ae20b17cbeaad31f1d59674f2a7b21ba002703a77e6c206136a5" Nov 03 21:48:08.296345 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:48:08.295788 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"294a95b54fb0ae20b17cbeaad31f1d59674f2a7b21ba002703a77e6c206136a5\": container with ID starting with 294a95b54fb0ae20b17cbeaad31f1d59674f2a7b21ba002703a77e6c206136a5 not found: ID does not exist" containerID="294a95b54fb0ae20b17cbeaad31f1d59674f2a7b21ba002703a77e6c206136a5" Nov 03 21:48:08.296345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.295851 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"294a95b54fb0ae20b17cbeaad31f1d59674f2a7b21ba002703a77e6c206136a5"} err="failed to get container status \"294a95b54fb0ae20b17cbeaad31f1d59674f2a7b21ba002703a77e6c206136a5\": rpc error: code = NotFound desc = could not find container \"294a95b54fb0ae20b17cbeaad31f1d59674f2a7b21ba002703a77e6c206136a5\": container with ID starting with 294a95b54fb0ae20b17cbeaad31f1d59674f2a7b21ba002703a77e6c206136a5 not found: ID does not exist" Nov 03 21:48:08.296345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.295876 5041 scope.go:122] "RemoveContainer" containerID="24a652966f936844adcfac8963d00e9fb6751cc2993ee5a29dd0344e8236fa58" Nov 03 21:48:08.296345 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:48:08.296227 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a652966f936844adcfac8963d00e9fb6751cc2993ee5a29dd0344e8236fa58\": container with ID starting with 24a652966f936844adcfac8963d00e9fb6751cc2993ee5a29dd0344e8236fa58 not found: ID does not exist" containerID="24a652966f936844adcfac8963d00e9fb6751cc2993ee5a29dd0344e8236fa58" Nov 03 21:48:08.296345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.296263 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a652966f936844adcfac8963d00e9fb6751cc2993ee5a29dd0344e8236fa58"} err="failed to get container status \"24a652966f936844adcfac8963d00e9fb6751cc2993ee5a29dd0344e8236fa58\": rpc error: code = NotFound desc = could not find container \"24a652966f936844adcfac8963d00e9fb6751cc2993ee5a29dd0344e8236fa58\": container with ID starting with 24a652966f936844adcfac8963d00e9fb6751cc2993ee5a29dd0344e8236fa58 not found: ID does not exist" Nov 03 21:48:08.296345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.296333 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-workload-socket\") pod \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " Nov 03 21:48:08.296596 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.296400 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-kube-api-access-6fq4l\" (UniqueName: \"kubernetes.io/projected/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-kube-api-access-6fq4l\") pod \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " Nov 03 21:48:08.296596 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.296498 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-podinfo\") pod \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " Nov 03 21:48:08.296656 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.296578 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-credential-socket\") pod \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " Nov 03 21:48:08.296656 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.296630 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-workload-certs\") pod \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " Nov 03 21:48:08.296656 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.296708 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istiod-ca-cert\") pod \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " Nov 03 21:48:08.299697 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.297145 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istiod-ca-cert" pod "4dcc2293-9b7c-4381-92f2-1657ee9a1e29" (UID: "4dcc2293-9b7c-4381-92f2-1657ee9a1e29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:08.299697 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.297359 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-token\" (UniqueName: \"kubernetes.io/projected/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-token\") pod \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " Nov 03 21:48:08.299697 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.297532 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-workload-socket" pod "4dcc2293-9b7c-4381-92f2-1657ee9a1e29" (UID: "4dcc2293-9b7c-4381-92f2-1657ee9a1e29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:08.299697 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.298048 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-envoy\") pod \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " Nov 03 21:48:08.299697 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.298098 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-data\") pod \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\" (UID: \"4dcc2293-9b7c-4381-92f2-1657ee9a1e29\") " Nov 03 21:48:08.299697 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.298940 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.299697 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.298965 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.299697 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.299374 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-credential-socket" pod "4dcc2293-9b7c-4381-92f2-1657ee9a1e29" (UID: "4dcc2293-9b7c-4381-92f2-1657ee9a1e29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:08.300824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.300767 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-data" pod "4dcc2293-9b7c-4381-92f2-1657ee9a1e29" (UID: "4dcc2293-9b7c-4381-92f2-1657ee9a1e29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:08.301207 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.301180 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-workload-certs" pod "4dcc2293-9b7c-4381-92f2-1657ee9a1e29" (UID: "4dcc2293-9b7c-4381-92f2-1657ee9a1e29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:08.302387 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.302325 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-podinfo" pod "4dcc2293-9b7c-4381-92f2-1657ee9a1e29" (UID: "4dcc2293-9b7c-4381-92f2-1657ee9a1e29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:08.304178 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.304156 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-kube-api-access-6fq4l" pod "4dcc2293-9b7c-4381-92f2-1657ee9a1e29" (UID: "4dcc2293-9b7c-4381-92f2-1657ee9a1e29"). InnerVolumeSpecName "kube-api-access-6fq4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:08.305290 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.305252 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-token" pod "4dcc2293-9b7c-4381-92f2-1657ee9a1e29" (UID: "4dcc2293-9b7c-4381-92f2-1657ee9a1e29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:08.307234 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.307202 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-envoy" pod "4dcc2293-9b7c-4381-92f2-1657ee9a1e29" (UID: "4dcc2293-9b7c-4381-92f2-1657ee9a1e29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:08.363681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.358414 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" podStartSLOduration=2.35839242 podStartE2EDuration="2.35839242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:08.344956237 +0000 UTC m=+1103.852184518" watchObservedRunningTime="2026-11-03 21:48:08.35839242 +0000 UTC m=+1103.865620701" Nov 03 21:48:08.400240 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.400210 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fq4l\" (UniqueName: \"kubernetes.io/projected/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-kube-api-access-6fq4l\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.400240 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.400240 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.400402 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.400251 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.400402 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.400262 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.400402 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.400270 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.400402 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.400278 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.400402 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.400285 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dcc2293-9b7c-4381-92f2-1657ee9a1e29-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.424427 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.424397 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:48:08.439414 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.439342 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:48:08.501113 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501074 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba2a45a-0811-49ae-84db-468f7950fbb6-kube-api-access-dt8gd\" (UniqueName: \"kubernetes.io/projected/5ba2a45a-0811-49ae-84db-468f7950fbb6-kube-api-access-dt8gd\") pod \"5ba2a45a-0811-49ae-84db-468f7950fbb6\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " Nov 03 21:48:08.501325 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501128 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6132e6c-086e-47bc-9baa-4a341f85671c-kube-api-access-j7qh9\" (UniqueName: \"kubernetes.io/projected/d6132e6c-086e-47bc-9baa-4a341f85671c-kube-api-access-j7qh9\") pod \"d6132e6c-086e-47bc-9baa-4a341f85671c\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " Nov 03 21:48:08.501325 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501161 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-token\" (UniqueName: \"kubernetes.io/projected/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-token\") pod \"d6132e6c-086e-47bc-9baa-4a341f85671c\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " Nov 03 21:48:08.501325 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501212 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-podinfo\") pod \"d6132e6c-086e-47bc-9baa-4a341f85671c\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " Nov 03 21:48:08.501325 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501246 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-workload-certs\") pod \"d6132e6c-086e-47bc-9baa-4a341f85671c\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " Nov 03 21:48:08.501548 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501348 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-data\") pod \"d6132e6c-086e-47bc-9baa-4a341f85671c\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " Nov 03 21:48:08.501548 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501406 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-envoy\") pod \"5ba2a45a-0811-49ae-84db-468f7950fbb6\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " Nov 03 21:48:08.501548 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501445 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-token\" (UniqueName: \"kubernetes.io/projected/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-token\") pod \"5ba2a45a-0811-49ae-84db-468f7950fbb6\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " Nov 03 21:48:08.501548 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501488 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-credential-socket\") pod \"d6132e6c-086e-47bc-9baa-4a341f85671c\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " Nov 03 21:48:08.502320 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501570 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-workload-certs\") pod \"5ba2a45a-0811-49ae-84db-468f7950fbb6\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " Nov 03 21:48:08.502320 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501630 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-envoy\") pod \"d6132e6c-086e-47bc-9baa-4a341f85671c\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " Nov 03 21:48:08.502320 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501698 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-credential-socket\") pod \"5ba2a45a-0811-49ae-84db-468f7950fbb6\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " Nov 03 21:48:08.502320 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501737 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-podinfo\") pod \"5ba2a45a-0811-49ae-84db-468f7950fbb6\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " Nov 03 21:48:08.502320 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501790 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6132e6c-086e-47bc-9baa-4a341f85671c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6132e6c-086e-47bc-9baa-4a341f85671c-istiod-ca-cert\") pod \"d6132e6c-086e-47bc-9baa-4a341f85671c\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " Nov 03 21:48:08.502320 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501829 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-data\") pod \"5ba2a45a-0811-49ae-84db-468f7950fbb6\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " Nov 03 21:48:08.502320 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501922 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-workload-socket\") pod \"5ba2a45a-0811-49ae-84db-468f7950fbb6\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " Nov 03 21:48:08.502320 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.501965 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-workload-socket\") pod \"d6132e6c-086e-47bc-9baa-4a341f85671c\" (UID: \"d6132e6c-086e-47bc-9baa-4a341f85671c\") " Nov 03 21:48:08.502320 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.502009 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba2a45a-0811-49ae-84db-468f7950fbb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba2a45a-0811-49ae-84db-468f7950fbb6-istiod-ca-cert\") pod \"5ba2a45a-0811-49ae-84db-468f7950fbb6\" (UID: \"5ba2a45a-0811-49ae-84db-468f7950fbb6\") " Nov 03 21:48:08.502823 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.502381 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-workload-socket" pod "5ba2a45a-0811-49ae-84db-468f7950fbb6" (UID: "5ba2a45a-0811-49ae-84db-468f7950fbb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:08.502823 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.502396 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-data" pod "5ba2a45a-0811-49ae-84db-468f7950fbb6" (UID: "5ba2a45a-0811-49ae-84db-468f7950fbb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:08.502823 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.502743 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6132e6c-086e-47bc-9baa-4a341f85671c-istiod-ca-cert" pod "d6132e6c-086e-47bc-9baa-4a341f85671c" (UID: "d6132e6c-086e-47bc-9baa-4a341f85671c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:08.502823 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.502746 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba2a45a-0811-49ae-84db-468f7950fbb6-istiod-ca-cert" pod "5ba2a45a-0811-49ae-84db-468f7950fbb6" (UID: "5ba2a45a-0811-49ae-84db-468f7950fbb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:08.502823 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.502796 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-workload-certs" pod "d6132e6c-086e-47bc-9baa-4a341f85671c" (UID: "d6132e6c-086e-47bc-9baa-4a341f85671c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:08.503077 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.502977 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-workload-certs" pod "5ba2a45a-0811-49ae-84db-468f7950fbb6" (UID: "5ba2a45a-0811-49ae-84db-468f7950fbb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:08.503077 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.503008 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-workload-socket" pod "d6132e6c-086e-47bc-9baa-4a341f85671c" (UID: "d6132e6c-086e-47bc-9baa-4a341f85671c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:08.503214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.503183 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-credential-socket" pod "5ba2a45a-0811-49ae-84db-468f7950fbb6" (UID: "5ba2a45a-0811-49ae-84db-468f7950fbb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:08.503395 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.503378 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.503472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.503398 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.503472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.503413 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6132e6c-086e-47bc-9baa-4a341f85671c-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.503472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.503427 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.503472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.503440 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.503472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.503453 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.503472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.503456 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-credential-socket" pod "d6132e6c-086e-47bc-9baa-4a341f85671c" (UID: "d6132e6c-086e-47bc-9baa-4a341f85671c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:08.503472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.503467 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba2a45a-0811-49ae-84db-468f7950fbb6-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.503927 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.503482 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.503927 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.503858 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-data" pod "d6132e6c-086e-47bc-9baa-4a341f85671c" (UID: "d6132e6c-086e-47bc-9baa-4a341f85671c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:08.504140 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.504114 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6132e6c-086e-47bc-9baa-4a341f85671c-kube-api-access-j7qh9" pod "d6132e6c-086e-47bc-9baa-4a341f85671c" (UID: "d6132e6c-086e-47bc-9baa-4a341f85671c"). InnerVolumeSpecName "kube-api-access-j7qh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:08.505826 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.505310 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-token" pod "d6132e6c-086e-47bc-9baa-4a341f85671c" (UID: "d6132e6c-086e-47bc-9baa-4a341f85671c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:08.505826 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.505626 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-envoy" pod "5ba2a45a-0811-49ae-84db-468f7950fbb6" (UID: "5ba2a45a-0811-49ae-84db-468f7950fbb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:08.505826 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.505796 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-podinfo" pod "5ba2a45a-0811-49ae-84db-468f7950fbb6" (UID: "5ba2a45a-0811-49ae-84db-468f7950fbb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:08.506802 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.506778 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-token" pod "5ba2a45a-0811-49ae-84db-468f7950fbb6" (UID: "5ba2a45a-0811-49ae-84db-468f7950fbb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:08.507017 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.506986 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-envoy" pod "d6132e6c-086e-47bc-9baa-4a341f85671c" (UID: "d6132e6c-086e-47bc-9baa-4a341f85671c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:08.507117 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.507034 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-podinfo" pod "d6132e6c-086e-47bc-9baa-4a341f85671c" (UID: "d6132e6c-086e-47bc-9baa-4a341f85671c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:08.507965 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.507939 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba2a45a-0811-49ae-84db-468f7950fbb6-kube-api-access-dt8gd" pod "5ba2a45a-0811-49ae-84db-468f7950fbb6" (UID: "5ba2a45a-0811-49ae-84db-468f7950fbb6"). InnerVolumeSpecName "kube-api-access-dt8gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:08.602935 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.602903 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h"] Nov 03 21:48:08.605396 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.605190 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt8gd\" (UniqueName: \"kubernetes.io/projected/5ba2a45a-0811-49ae-84db-468f7950fbb6-kube-api-access-dt8gd\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.605396 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.605226 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7qh9\" (UniqueName: \"kubernetes.io/projected/d6132e6c-086e-47bc-9baa-4a341f85671c-kube-api-access-j7qh9\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.605396 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.605241 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.605396 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.605257 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.605396 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.605315 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.605396 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.605329 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.605396 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.605342 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.605396 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.605354 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.605396 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.605366 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6132e6c-086e-47bc-9baa-4a341f85671c-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.605396 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.605379 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba2a45a-0811-49ae-84db-468f7950fbb6-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:08.611638 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.611592 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-7rh4h"] Nov 03 21:48:08.627913 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.627866 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" path="/var/lib/kubelet/pods/4dcc2293-9b7c-4381-92f2-1657ee9a1e29/volumes" Nov 03 21:48:08.964112 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:08.964074 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" containerID="cri-o://d2587b4dd0fd47ef23d48b30df1bf0a07d494e9451beb3fa77b73fa71aa1ed35" gracePeriod=45 Nov 03 21:48:09.285979 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.285942 5041 generic.go:356] "Generic (PLEG): container finished" podID="3a8a6d82-56a3-4ef0-9785-f9578579b5f0" containerID="d4acb8a9d294b980b519a5b62cb67f8ef3ee45e583e4746a15ce01a1914ad428" exitCode=0 Nov 03 21:48:09.286371 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.286018 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" event={"ID":"3a8a6d82-56a3-4ef0-9785-f9578579b5f0","Type":"ContainerDied","Data":"d4acb8a9d294b980b519a5b62cb67f8ef3ee45e583e4746a15ce01a1914ad428"} Nov 03 21:48:09.289455 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.289426 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" Nov 03 21:48:09.289608 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.289469 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n" event={"ID":"5ba2a45a-0811-49ae-84db-468f7950fbb6","Type":"ContainerDied","Data":"185241c1dcb38732cd546a7a779683fbf7b41e0f92850935cdec3f854fe71054"} Nov 03 21:48:09.289608 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.289536 5041 scope.go:122] "RemoveContainer" containerID="1a23f6899f664aa83630f2f57214edf337bfe9c30801ab7695962debf8dd89fe" Nov 03 21:48:09.296087 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.296051 5041 generic.go:356] "Generic (PLEG): container finished" podID="3f52ca60-4d7e-4707-874a-02632382a76d" containerID="d2587b4dd0fd47ef23d48b30df1bf0a07d494e9451beb3fa77b73fa71aa1ed35" exitCode=0 Nov 03 21:48:09.296386 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.296364 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" event={"ID":"3f52ca60-4d7e-4707-874a-02632382a76d","Type":"ContainerDied","Data":"d2587b4dd0fd47ef23d48b30df1bf0a07d494e9451beb3fa77b73fa71aa1ed35"} Nov 03 21:48:09.296503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.296410 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" event={"ID":"3f52ca60-4d7e-4707-874a-02632382a76d","Type":"ContainerDied","Data":"4df39328d08e0b0abaeac3d51c2ab5d9b4311cd865b161f087bbcf825540f454"} Nov 03 21:48:09.296503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.296428 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4df39328d08e0b0abaeac3d51c2ab5d9b4311cd865b161f087bbcf825540f454" Nov 03 21:48:09.298724 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.298642 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" event={"ID":"d6132e6c-086e-47bc-9baa-4a341f85671c","Type":"ContainerDied","Data":"bf1d5883134a927b884aacc6dcb281ada25376461c0800c6caeb8674cc55cab0"} Nov 03 21:48:09.298862 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.298825 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd" Nov 03 21:48:09.358613 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.358586 5041 scope.go:122] "RemoveContainer" containerID="320a8f899cc06d57d818f2b5f37146911aff2531e13003dcee0188e63603562e" Nov 03 21:48:09.381736 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.377952 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:48:09.433580 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.433544 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-model-cache\") pod \"3f52ca60-4d7e-4707-874a-02632382a76d\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " Nov 03 21:48:09.433786 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.433616 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f52ca60-4d7e-4707-874a-02632382a76d-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f52ca60-4d7e-4707-874a-02632382a76d-tls-certs\") pod \"3f52ca60-4d7e-4707-874a-02632382a76d\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " Nov 03 21:48:09.433887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.433871 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f52ca60-4d7e-4707-874a-02632382a76d-kube-api-access-vf2qp\" (UniqueName: \"kubernetes.io/projected/3f52ca60-4d7e-4707-874a-02632382a76d-kube-api-access-vf2qp\") pod \"3f52ca60-4d7e-4707-874a-02632382a76d\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " Nov 03 21:48:09.433945 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.433907 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-dshm\") pod \"3f52ca60-4d7e-4707-874a-02632382a76d\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " Nov 03 21:48:09.433945 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.433915 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-model-cache" pod "3f52ca60-4d7e-4707-874a-02632382a76d" (UID: "3f52ca60-4d7e-4707-874a-02632382a76d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:09.434046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.434031 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-tmp-dir\") pod \"3f52ca60-4d7e-4707-874a-02632382a76d\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " Nov 03 21:48:09.434097 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.434082 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-home\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-home\") pod \"3f52ca60-4d7e-4707-874a-02632382a76d\" (UID: \"3f52ca60-4d7e-4707-874a-02632382a76d\") " Nov 03 21:48:09.434513 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.434484 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-tmp-dir" pod "3f52ca60-4d7e-4707-874a-02632382a76d" (UID: "3f52ca60-4d7e-4707-874a-02632382a76d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:09.434708 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.434686 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-home" pod "3f52ca60-4d7e-4707-874a-02632382a76d" (UID: "3f52ca60-4d7e-4707-874a-02632382a76d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:09.434916 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.434891 5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-tmp-dir\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:09.434978 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.434922 5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-home\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:09.434978 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.434938 5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-model-cache\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:09.436967 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.436939 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f52ca60-4d7e-4707-874a-02632382a76d-kube-api-access-vf2qp" pod "3f52ca60-4d7e-4707-874a-02632382a76d" (UID: "3f52ca60-4d7e-4707-874a-02632382a76d"). InnerVolumeSpecName "kube-api-access-vf2qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:09.440066 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.438961 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-dshm" pod "3f52ca60-4d7e-4707-874a-02632382a76d" (UID: "3f52ca60-4d7e-4707-874a-02632382a76d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:09.441723 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.441551 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f52ca60-4d7e-4707-874a-02632382a76d-tls-certs" pod "3f52ca60-4d7e-4707-874a-02632382a76d" (UID: "3f52ca60-4d7e-4707-874a-02632382a76d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:09.492789 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.492745 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd"] Nov 03 21:48:09.499402 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.499364 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-bbxpd"] Nov 03 21:48:09.531472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.531426 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n"] Nov 03 21:48:09.536954 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.536674 5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f52ca60-4d7e-4707-874a-02632382a76d-tls-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:09.536954 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.536718 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf2qp\" (UniqueName: \"kubernetes.io/projected/3f52ca60-4d7e-4707-874a-02632382a76d-kube-api-access-vf2qp\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:09.536954 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.536734 5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f52ca60-4d7e-4707-874a-02632382a76d-dshm\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:09.541702 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:09.541646 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5620-openshift-default-86d5bc584f-4bn8n"] Nov 03 21:48:10.088238 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.088198 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" podUID="f05ce451-3614-4af6-88e8-8793cf631ea1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:48:10.312474 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.312437 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf" Nov 03 21:48:10.373415 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.373327 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf"] Nov 03 21:48:10.391279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.391119 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-05bc27/distinct-2-w4-05bc27-kserve-5879bfbd9c-7wgqf"] Nov 03 21:48:10.512021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.511982 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" Nov 03 21:48:10.558911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.558875 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-kube-api-access-7vrkf\" (UniqueName: \"kubernetes.io/projected/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-kube-api-access-7vrkf\") pod \"3a8a6d82-56a3-4ef0-9785-f9578579b5f0\" (UID: \"3a8a6d82-56a3-4ef0-9785-f9578579b5f0\") " Nov 03 21:48:10.559102 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.558928 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-maas-api-openshift-ca\") pod \"3a8a6d82-56a3-4ef0-9785-f9578579b5f0\" (UID: \"3a8a6d82-56a3-4ef0-9785-f9578579b5f0\") " Nov 03 21:48:10.559240 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.559220 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-maas-api-xks-ca\") pod \"3a8a6d82-56a3-4ef0-9785-f9578579b5f0\" (UID: \"3a8a6d82-56a3-4ef0-9785-f9578579b5f0\") " Nov 03 21:48:10.559459 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.559406 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-maas-api-openshift-ca" pod "3a8a6d82-56a3-4ef0-9785-f9578579b5f0" (UID: "3a8a6d82-56a3-4ef0-9785-f9578579b5f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:10.559968 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.559942 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:10.561509 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.561478 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-kube-api-access-7vrkf" pod "3a8a6d82-56a3-4ef0-9785-f9578579b5f0" (UID: "3a8a6d82-56a3-4ef0-9785-f9578579b5f0"). InnerVolumeSpecName "kube-api-access-7vrkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:10.562837 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.561886 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-maas-api-xks-ca" pod "3a8a6d82-56a3-4ef0-9785-f9578579b5f0" (UID: "3a8a6d82-56a3-4ef0-9785-f9578579b5f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:10.596511 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.596472 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt"] Nov 03 21:48:10.597303 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597268 5041 cpu_manager.go:414] "containerMap: removing container" podUID="d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93" containerName="revoke-keys" Nov 03 21:48:10.597303 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597299 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93" containerName="revoke-keys" Nov 03 21:48:10.597504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597321 5041 cpu_manager.go:414] "containerMap: removing container" podUID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" containerName="istio-proxy" Nov 03 21:48:10.597504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597327 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" containerName="istio-proxy" Nov 03 21:48:10.597504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597334 5041 cpu_manager.go:414] "containerMap: removing container" podUID="76d6f982-faed-4a04-aa25-85a2159c343b" containerName="revoke-keys" Nov 03 21:48:10.597504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597339 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d6f982-faed-4a04-aa25-85a2159c343b" containerName="revoke-keys" Nov 03 21:48:10.597504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597366 5041 cpu_manager.go:414] "containerMap: removing container" podUID="3a8a6d82-56a3-4ef0-9785-f9578579b5f0" containerName="revoke-keys" Nov 03 21:48:10.597504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597373 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a8a6d82-56a3-4ef0-9785-f9578579b5f0" containerName="revoke-keys" Nov 03 21:48:10.597504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597398 5041 cpu_manager.go:414] "containerMap: removing container" podUID="d6132e6c-086e-47bc-9baa-4a341f85671c" containerName="istio-proxy" Nov 03 21:48:10.597504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597404 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6132e6c-086e-47bc-9baa-4a341f85671c" containerName="istio-proxy" Nov 03 21:48:10.597504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597414 5041 cpu_manager.go:414] "containerMap: removing container" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" Nov 03 21:48:10.597504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597418 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" Nov 03 21:48:10.597504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597431 5041 cpu_manager.go:414] "containerMap: removing container" podUID="5ba2a45a-0811-49ae-84db-468f7950fbb6" containerName="istio-proxy" Nov 03 21:48:10.597504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597436 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba2a45a-0811-49ae-84db-468f7950fbb6" containerName="istio-proxy" Nov 03 21:48:10.597504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597451 5041 cpu_manager.go:414] "containerMap: removing container" podUID="56f38640-6dc3-47b9-8e3b-36efdbd57c68" containerName="revoke-keys" Nov 03 21:48:10.597504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597456 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f38640-6dc3-47b9-8e3b-36efdbd57c68" containerName="revoke-keys" Nov 03 21:48:10.598215 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597561 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6132e6c-086e-47bc-9baa-4a341f85671c" containerName="istio-proxy" Nov 03 21:48:10.598215 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597571 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" containerName="main" Nov 03 21:48:10.598215 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597578 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" containerName="istio-proxy" Nov 03 21:48:10.598215 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597584 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba2a45a-0811-49ae-84db-468f7950fbb6" containerName="istio-proxy" Nov 03 21:48:10.598215 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597591 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f38640-6dc3-47b9-8e3b-36efdbd57c68" containerName="revoke-keys" Nov 03 21:48:10.598215 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597599 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93" containerName="revoke-keys" Nov 03 21:48:10.598215 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597606 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" containerName="istio-proxy" Nov 03 21:48:10.598215 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597612 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d6f982-faed-4a04-aa25-85a2159c343b" containerName="revoke-keys" Nov 03 21:48:10.598215 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.597625 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a8a6d82-56a3-4ef0-9785-f9578579b5f0" containerName="revoke-keys" Nov 03 21:48:10.642281 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.639107 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt" Nov 03 21:48:10.651008 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.650970 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f52ca60-4d7e-4707-874a-02632382a76d" path="/var/lib/kubelet/pods/3f52ca60-4d7e-4707-874a-02632382a76d/volumes" Nov 03 21:48:10.651674 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.651637 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba2a45a-0811-49ae-84db-468f7950fbb6" path="/var/lib/kubelet/pods/5ba2a45a-0811-49ae-84db-468f7950fbb6/volumes" Nov 03 21:48:10.652253 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.652234 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6132e6c-086e-47bc-9baa-4a341f85671c" path="/var/lib/kubelet/pods/d6132e6c-086e-47bc-9baa-4a341f85671c/volumes" Nov 03 21:48:10.653296 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.653272 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt"] Nov 03 21:48:10.653750 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.653465 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4"] Nov 03 21:48:10.655208 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.655184 5041 cpu_manager.go:414] "containerMap: removing container" podUID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" containerName="istio-proxy" Nov 03 21:48:10.655208 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.655209 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dcc2293-9b7c-4381-92f2-1657ee9a1e29" containerName="istio-proxy" Nov 03 21:48:10.662254 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.662222 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vrkf\" (UniqueName: \"kubernetes.io/projected/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-kube-api-access-7vrkf\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:10.662399 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.662257 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a8a6d82-56a3-4ef0-9785-f9578579b5f0-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:10.664062 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.664036 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4"] Nov 03 21:48:10.664208 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.664151 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4" Nov 03 21:48:10.763490 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.763451 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44581ba9-5165-4cbd-91ea-3f53777faf74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c787ced1-7tcjt\" (UID: \"44581ba9-5165-4cbd-91ea-3f53777faf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt" Nov 03 21:48:10.763490 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.763494 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh7h7\" (UniqueName: \"kubernetes.io/projected/3ff7c5c1-0631-4596-ac51-a0578f81953c-kube-api-access-vh7h7\") pod \"maas-api-revoke-sub-8c21b054-zq8l4\" (UID: \"3ff7c5c1-0631-4596-ac51-a0578f81953c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4" Nov 03 21:48:10.763758 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.763520 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff7c5c1-0631-4596-ac51-a0578f81953c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c21b054-zq8l4\" (UID: \"3ff7c5c1-0631-4596-ac51-a0578f81953c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4" Nov 03 21:48:10.763831 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.763757 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44hcm\" (UniqueName: \"kubernetes.io/projected/44581ba9-5165-4cbd-91ea-3f53777faf74-kube-api-access-44hcm\") pod \"maas-api-revoke-sub-c787ced1-7tcjt\" (UID: \"44581ba9-5165-4cbd-91ea-3f53777faf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt" Nov 03 21:48:10.763909 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.763895 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44581ba9-5165-4cbd-91ea-3f53777faf74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c787ced1-7tcjt\" (UID: \"44581ba9-5165-4cbd-91ea-3f53777faf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt" Nov 03 21:48:10.764329 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.764291 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff7c5c1-0631-4596-ac51-a0578f81953c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c21b054-zq8l4\" (UID: \"3ff7c5c1-0631-4596-ac51-a0578f81953c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4" Nov 03 21:48:10.779821 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:48:10.779784 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:48:10.866418 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.866380 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff7c5c1-0631-4596-ac51-a0578f81953c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c21b054-zq8l4\" (UID: \"3ff7c5c1-0631-4596-ac51-a0578f81953c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4" Nov 03 21:48:10.866586 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.866462 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44581ba9-5165-4cbd-91ea-3f53777faf74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c787ced1-7tcjt\" (UID: \"44581ba9-5165-4cbd-91ea-3f53777faf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt" Nov 03 21:48:10.866586 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.866486 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7h7\" (UniqueName: \"kubernetes.io/projected/3ff7c5c1-0631-4596-ac51-a0578f81953c-kube-api-access-vh7h7\") pod \"maas-api-revoke-sub-8c21b054-zq8l4\" (UID: \"3ff7c5c1-0631-4596-ac51-a0578f81953c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4" Nov 03 21:48:10.866586 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.866508 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff7c5c1-0631-4596-ac51-a0578f81953c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c21b054-zq8l4\" (UID: \"3ff7c5c1-0631-4596-ac51-a0578f81953c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4" Nov 03 21:48:10.866801 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.866639 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44hcm\" (UniqueName: \"kubernetes.io/projected/44581ba9-5165-4cbd-91ea-3f53777faf74-kube-api-access-44hcm\") pod \"maas-api-revoke-sub-c787ced1-7tcjt\" (UID: \"44581ba9-5165-4cbd-91ea-3f53777faf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt" Nov 03 21:48:10.866801 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.866790 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44581ba9-5165-4cbd-91ea-3f53777faf74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c787ced1-7tcjt\" (UID: \"44581ba9-5165-4cbd-91ea-3f53777faf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt" Nov 03 21:48:10.867329 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.867303 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff7c5c1-0631-4596-ac51-a0578f81953c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c21b054-zq8l4\" (UID: \"3ff7c5c1-0631-4596-ac51-a0578f81953c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4" Nov 03 21:48:10.867516 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.867495 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44581ba9-5165-4cbd-91ea-3f53777faf74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c787ced1-7tcjt\" (UID: \"44581ba9-5165-4cbd-91ea-3f53777faf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt" Nov 03 21:48:10.869071 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.869044 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff7c5c1-0631-4596-ac51-a0578f81953c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c21b054-zq8l4\" (UID: \"3ff7c5c1-0631-4596-ac51-a0578f81953c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4" Nov 03 21:48:10.869071 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.869066 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44581ba9-5165-4cbd-91ea-3f53777faf74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c787ced1-7tcjt\" (UID: \"44581ba9-5165-4cbd-91ea-3f53777faf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt" Nov 03 21:48:10.887013 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.886981 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh7h7\" (UniqueName: \"kubernetes.io/projected/3ff7c5c1-0631-4596-ac51-a0578f81953c-kube-api-access-vh7h7\") pod \"maas-api-revoke-sub-8c21b054-zq8l4\" (UID: \"3ff7c5c1-0631-4596-ac51-a0578f81953c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4" Nov 03 21:48:10.887153 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.887028 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44hcm\" (UniqueName: \"kubernetes.io/projected/44581ba9-5165-4cbd-91ea-3f53777faf74-kube-api-access-44hcm\") pod \"maas-api-revoke-sub-c787ced1-7tcjt\" (UID: \"44581ba9-5165-4cbd-91ea-3f53777faf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt" Nov 03 21:48:10.973967 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.973925 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt" Nov 03 21:48:10.990885 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:10.990843 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4" Nov 03 21:48:11.286921 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:48:11.286877 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44581ba9_5165_4cbd_91ea_3f53777faf74.slice/crio-947b4967d5658157db071b025d88a3c7b8a69467535c81573fc5401517b9fcb0 WatchSource:0}: Error finding container 947b4967d5658157db071b025d88a3c7b8a69467535c81573fc5401517b9fcb0: Status 404 returned error can't find the container with id 947b4967d5658157db071b025d88a3c7b8a69467535c81573fc5401517b9fcb0 Nov 03 21:48:11.290563 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:11.290537 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt"] Nov 03 21:48:11.307921 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:48:11.307874 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff7c5c1_0631_4596_ac51_a0578f81953c.slice/crio-bc2089b1995da9fe95f4db140e2396daf720c83d4e586ba0f2c0b21dc5a21313 WatchSource:0}: Error finding container bc2089b1995da9fe95f4db140e2396daf720c83d4e586ba0f2c0b21dc5a21313: Status 404 returned error can't find the container with id bc2089b1995da9fe95f4db140e2396daf720c83d4e586ba0f2c0b21dc5a21313 Nov 03 21:48:11.310538 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:11.310514 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4"] Nov 03 21:48:11.324313 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:11.324280 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4" event={"ID":"3ff7c5c1-0631-4596-ac51-a0578f81953c","Type":"ContainerStarted","Data":"bc2089b1995da9fe95f4db140e2396daf720c83d4e586ba0f2c0b21dc5a21313"} Nov 03 21:48:11.325976 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:11.325947 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt" event={"ID":"44581ba9-5165-4cbd-91ea-3f53777faf74","Type":"ContainerStarted","Data":"947b4967d5658157db071b025d88a3c7b8a69467535c81573fc5401517b9fcb0"} Nov 03 21:48:11.327829 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:11.327692 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" event={"ID":"3a8a6d82-56a3-4ef0-9785-f9578579b5f0","Type":"ContainerDied","Data":"5b5cd9e2f1ed25eb9ecf4f57118c72ab6dab27a4e0d88fd207a24e9055e6e6b0"} Nov 03 21:48:11.327829 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:11.327734 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b5cd9e2f1ed25eb9ecf4f57118c72ab6dab27a4e0d88fd207a24e9055e6e6b0" Nov 03 21:48:11.327829 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:11.327733 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql" Nov 03 21:48:12.339130 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:12.339093 5041 generic.go:356] "Generic (PLEG): container finished" podID="44581ba9-5165-4cbd-91ea-3f53777faf74" containerID="16747b27e60ebc8489a94b02ab0ad34196a5e53d2752b1bfe39ba19b7dac509b" exitCode=0 Nov 03 21:48:12.339576 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:12.339171 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt" event={"ID":"44581ba9-5165-4cbd-91ea-3f53777faf74","Type":"ContainerDied","Data":"16747b27e60ebc8489a94b02ab0ad34196a5e53d2752b1bfe39ba19b7dac509b"} Nov 03 21:48:12.341583 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:12.341554 5041 generic.go:356] "Generic (PLEG): container finished" podID="3ff7c5c1-0631-4596-ac51-a0578f81953c" containerID="38763417e5116db28b61ea25b76a4b1d8a9a833ef7829c45edffe7bdcf65f749" exitCode=0 Nov 03 21:48:12.341759 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:12.341641 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4" event={"ID":"3ff7c5c1-0631-4596-ac51-a0578f81953c","Type":"ContainerDied","Data":"38763417e5116db28b61ea25b76a4b1d8a9a833ef7829c45edffe7bdcf65f749"} Nov 03 21:48:13.589836 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.589800 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt" Nov 03 21:48:13.638195 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.638104 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4" Nov 03 21:48:13.718187 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.718144 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ff7c5c1-0631-4596-ac51-a0578f81953c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff7c5c1-0631-4596-ac51-a0578f81953c-maas-api-xks-ca\") pod \"3ff7c5c1-0631-4596-ac51-a0578f81953c\" (UID: \"3ff7c5c1-0631-4596-ac51-a0578f81953c\") " Nov 03 21:48:13.718359 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.718264 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44581ba9-5165-4cbd-91ea-3f53777faf74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44581ba9-5165-4cbd-91ea-3f53777faf74-maas-api-openshift-ca\") pod \"44581ba9-5165-4cbd-91ea-3f53777faf74\" (UID: \"44581ba9-5165-4cbd-91ea-3f53777faf74\") " Nov 03 21:48:13.718500 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.718408 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff7c5c1-0631-4596-ac51-a0578f81953c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff7c5c1-0631-4596-ac51-a0578f81953c-maas-api-openshift-ca\") pod \"3ff7c5c1-0631-4596-ac51-a0578f81953c\" (UID: \"3ff7c5c1-0631-4596-ac51-a0578f81953c\") " Nov 03 21:48:13.718558 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.718526 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44581ba9-5165-4cbd-91ea-3f53777faf74-kube-api-access-44hcm\" (UniqueName: \"kubernetes.io/projected/44581ba9-5165-4cbd-91ea-3f53777faf74-kube-api-access-44hcm\") pod \"44581ba9-5165-4cbd-91ea-3f53777faf74\" (UID: \"44581ba9-5165-4cbd-91ea-3f53777faf74\") " Nov 03 21:48:13.718712 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.718625 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44581ba9-5165-4cbd-91ea-3f53777faf74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44581ba9-5165-4cbd-91ea-3f53777faf74-maas-api-xks-ca\") pod \"44581ba9-5165-4cbd-91ea-3f53777faf74\" (UID: \"44581ba9-5165-4cbd-91ea-3f53777faf74\") " Nov 03 21:48:13.718874 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.718739 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff7c5c1-0631-4596-ac51-a0578f81953c-kube-api-access-vh7h7\" (UniqueName: \"kubernetes.io/projected/3ff7c5c1-0631-4596-ac51-a0578f81953c-kube-api-access-vh7h7\") pod \"3ff7c5c1-0631-4596-ac51-a0578f81953c\" (UID: \"3ff7c5c1-0631-4596-ac51-a0578f81953c\") " Nov 03 21:48:13.720710 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.720635 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44581ba9-5165-4cbd-91ea-3f53777faf74-maas-api-openshift-ca" pod "44581ba9-5165-4cbd-91ea-3f53777faf74" (UID: "44581ba9-5165-4cbd-91ea-3f53777faf74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:13.721382 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.721352 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff7c5c1-0631-4596-ac51-a0578f81953c-maas-api-openshift-ca" pod "3ff7c5c1-0631-4596-ac51-a0578f81953c" (UID: "3ff7c5c1-0631-4596-ac51-a0578f81953c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:13.722773 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.722745 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ff7c5c1-0631-4596-ac51-a0578f81953c-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:13.723039 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.723015 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44581ba9-5165-4cbd-91ea-3f53777faf74-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:13.725915 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.725884 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44581ba9-5165-4cbd-91ea-3f53777faf74-kube-api-access-44hcm" pod "44581ba9-5165-4cbd-91ea-3f53777faf74" (UID: "44581ba9-5165-4cbd-91ea-3f53777faf74"). InnerVolumeSpecName "kube-api-access-44hcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:13.726068 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.726036 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44581ba9-5165-4cbd-91ea-3f53777faf74-maas-api-xks-ca" pod "44581ba9-5165-4cbd-91ea-3f53777faf74" (UID: "44581ba9-5165-4cbd-91ea-3f53777faf74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:13.726283 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.726258 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff7c5c1-0631-4596-ac51-a0578f81953c-kube-api-access-vh7h7" pod "3ff7c5c1-0631-4596-ac51-a0578f81953c" (UID: "3ff7c5c1-0631-4596-ac51-a0578f81953c"). InnerVolumeSpecName "kube-api-access-vh7h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:13.730101 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.730073 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff7c5c1-0631-4596-ac51-a0578f81953c-maas-api-xks-ca" pod "3ff7c5c1-0631-4596-ac51-a0578f81953c" (UID: "3ff7c5c1-0631-4596-ac51-a0578f81953c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:13.826290 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.826193 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh7h7\" (UniqueName: \"kubernetes.io/projected/3ff7c5c1-0631-4596-ac51-a0578f81953c-kube-api-access-vh7h7\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:13.826290 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.826231 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ff7c5c1-0631-4596-ac51-a0578f81953c-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:13.826290 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.826245 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44hcm\" (UniqueName: \"kubernetes.io/projected/44581ba9-5165-4cbd-91ea-3f53777faf74-kube-api-access-44hcm\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:13.826290 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:13.826258 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44581ba9-5165-4cbd-91ea-3f53777faf74-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:14.450687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:14.450498 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt" event={"ID":"44581ba9-5165-4cbd-91ea-3f53777faf74","Type":"ContainerDied","Data":"947b4967d5658157db071b025d88a3c7b8a69467535c81573fc5401517b9fcb0"} Nov 03 21:48:14.450687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:14.450609 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="947b4967d5658157db071b025d88a3c7b8a69467535c81573fc5401517b9fcb0" Nov 03 21:48:14.450920 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:14.450752 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt" Nov 03 21:48:14.484717 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:14.468400 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4" Nov 03 21:48:14.484717 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:14.468432 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4" event={"ID":"3ff7c5c1-0631-4596-ac51-a0578f81953c","Type":"ContainerDied","Data":"bc2089b1995da9fe95f4db140e2396daf720c83d4e586ba0f2c0b21dc5a21313"} Nov 03 21:48:14.484717 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:14.468500 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc2089b1995da9fe95f4db140e2396daf720c83d4e586ba0f2c0b21dc5a21313" Nov 03 21:48:17.061251 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:17.061206 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" podUID="cabaa235-b28f-41a4-ad06-a10e60adc05e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:48:18.269975 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.269934 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg"] Nov 03 21:48:18.271107 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.271069 5041 cpu_manager.go:414] "containerMap: removing container" podUID="3ff7c5c1-0631-4596-ac51-a0578f81953c" containerName="revoke-keys" Nov 03 21:48:18.271107 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.271099 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff7c5c1-0631-4596-ac51-a0578f81953c" containerName="revoke-keys" Nov 03 21:48:18.271311 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.271156 5041 cpu_manager.go:414] "containerMap: removing container" podUID="44581ba9-5165-4cbd-91ea-3f53777faf74" containerName="revoke-keys" Nov 03 21:48:18.271311 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.271166 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44581ba9-5165-4cbd-91ea-3f53777faf74" containerName="revoke-keys" Nov 03 21:48:18.271439 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.271369 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff7c5c1-0631-4596-ac51-a0578f81953c" containerName="revoke-keys" Nov 03 21:48:18.271439 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.271395 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="44581ba9-5165-4cbd-91ea-3f53777faf74" containerName="revoke-keys" Nov 03 21:48:18.276982 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.276951 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" Nov 03 21:48:18.385775 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.354367 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a894e07-8a02-471c-90f6-0dc4db7c25e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-885ff216-96mdg\" (UID: \"8a894e07-8a02-471c-90f6-0dc4db7c25e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" Nov 03 21:48:18.389684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.386195 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a894e07-8a02-471c-90f6-0dc4db7c25e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-885ff216-96mdg\" (UID: \"8a894e07-8a02-471c-90f6-0dc4db7c25e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" Nov 03 21:48:18.418687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.395123 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdxkp\" (UniqueName: \"kubernetes.io/projected/8a894e07-8a02-471c-90f6-0dc4db7c25e9-kube-api-access-rdxkp\") pod \"maas-api-revoke-sub-885ff216-96mdg\" (UID: \"8a894e07-8a02-471c-90f6-0dc4db7c25e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" Nov 03 21:48:18.483847 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.483812 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg"] Nov 03 21:48:18.539015 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.538891 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdxkp\" (UniqueName: \"kubernetes.io/projected/8a894e07-8a02-471c-90f6-0dc4db7c25e9-kube-api-access-rdxkp\") pod \"maas-api-revoke-sub-885ff216-96mdg\" (UID: \"8a894e07-8a02-471c-90f6-0dc4db7c25e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" Nov 03 21:48:18.551911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.551871 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a894e07-8a02-471c-90f6-0dc4db7c25e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-885ff216-96mdg\" (UID: \"8a894e07-8a02-471c-90f6-0dc4db7c25e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" Nov 03 21:48:18.552283 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.552260 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a894e07-8a02-471c-90f6-0dc4db7c25e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-885ff216-96mdg\" (UID: \"8a894e07-8a02-471c-90f6-0dc4db7c25e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" Nov 03 21:48:18.552819 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.545276 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:48:18.552950 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.545517 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:48:18.553044 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.552868 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a894e07-8a02-471c-90f6-0dc4db7c25e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-885ff216-96mdg\" (UID: \"8a894e07-8a02-471c-90f6-0dc4db7c25e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" Nov 03 21:48:18.591481 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.591446 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a894e07-8a02-471c-90f6-0dc4db7c25e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-885ff216-96mdg\" (UID: \"8a894e07-8a02-471c-90f6-0dc4db7c25e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" Nov 03 21:48:18.591760 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.591741 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z"] Nov 03 21:48:18.605899 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.605868 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z" Nov 03 21:48:18.622777 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.622646 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdxkp\" (UniqueName: \"kubernetes.io/projected/8a894e07-8a02-471c-90f6-0dc4db7c25e9-kube-api-access-rdxkp\") pod \"maas-api-revoke-sub-885ff216-96mdg\" (UID: \"8a894e07-8a02-471c-90f6-0dc4db7c25e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" Nov 03 21:48:18.649621 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.648937 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z"] Nov 03 21:48:18.654677 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.654625 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d975a235-5acc-47f6-9fc1-da6131f27b1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f80204f-h952z\" (UID: \"d975a235-5acc-47f6-9fc1-da6131f27b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z" Nov 03 21:48:18.654879 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.654833 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d975a235-5acc-47f6-9fc1-da6131f27b1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f80204f-h952z\" (UID: \"d975a235-5acc-47f6-9fc1-da6131f27b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z" Nov 03 21:48:18.655088 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.655064 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qphgt\" (UniqueName: \"kubernetes.io/projected/d975a235-5acc-47f6-9fc1-da6131f27b1b-kube-api-access-qphgt\") pod \"maas-api-revoke-sub-8f80204f-h952z\" (UID: \"d975a235-5acc-47f6-9fc1-da6131f27b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z" Nov 03 21:48:18.700883 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.700845 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" Nov 03 21:48:18.757715 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.757082 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d975a235-5acc-47f6-9fc1-da6131f27b1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f80204f-h952z\" (UID: \"d975a235-5acc-47f6-9fc1-da6131f27b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z" Nov 03 21:48:18.757715 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.757181 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d975a235-5acc-47f6-9fc1-da6131f27b1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f80204f-h952z\" (UID: \"d975a235-5acc-47f6-9fc1-da6131f27b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z" Nov 03 21:48:18.757715 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.757276 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qphgt\" (UniqueName: \"kubernetes.io/projected/d975a235-5acc-47f6-9fc1-da6131f27b1b-kube-api-access-qphgt\") pod \"maas-api-revoke-sub-8f80204f-h952z\" (UID: \"d975a235-5acc-47f6-9fc1-da6131f27b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z" Nov 03 21:48:18.758263 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.758238 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d975a235-5acc-47f6-9fc1-da6131f27b1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f80204f-h952z\" (UID: \"d975a235-5acc-47f6-9fc1-da6131f27b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z" Nov 03 21:48:18.760584 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.760529 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d975a235-5acc-47f6-9fc1-da6131f27b1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f80204f-h952z\" (UID: \"d975a235-5acc-47f6-9fc1-da6131f27b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z" Nov 03 21:48:18.781933 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.781885 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qphgt\" (UniqueName: \"kubernetes.io/projected/d975a235-5acc-47f6-9fc1-da6131f27b1b-kube-api-access-qphgt\") pod \"maas-api-revoke-sub-8f80204f-h952z\" (UID: \"d975a235-5acc-47f6-9fc1-da6131f27b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z" Nov 03 21:48:18.806547 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.805243 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9"] Nov 03 21:48:18.847811 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.847762 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4eced5-qgjf9"] Nov 03 21:48:18.967977 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:18.967938 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z" Nov 03 21:48:19.023973 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:19.023940 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg"] Nov 03 21:48:19.286480 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:48:19.286446 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd975a235_5acc_47f6_9fc1_da6131f27b1b.slice/crio-4d089e0deee3dfa9435cf5530cdf7b271674c094d7263eb46d7c87ebe61d2683 WatchSource:0}: Error finding container 4d089e0deee3dfa9435cf5530cdf7b271674c094d7263eb46d7c87ebe61d2683: Status 404 returned error can't find the container with id 4d089e0deee3dfa9435cf5530cdf7b271674c094d7263eb46d7c87ebe61d2683 Nov 03 21:48:19.288478 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:19.288451 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z"] Nov 03 21:48:19.593782 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:19.593684 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z" event={"ID":"d975a235-5acc-47f6-9fc1-da6131f27b1b","Type":"ContainerStarted","Data":"4d089e0deee3dfa9435cf5530cdf7b271674c094d7263eb46d7c87ebe61d2683"} Nov 03 21:48:19.595979 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:19.595947 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" event={"ID":"8a894e07-8a02-471c-90f6-0dc4db7c25e9","Type":"ContainerStarted","Data":"acb75c4939541f7360323c73831a67c3b88d7a8484d9f753bbbdf23012b1037e"} Nov 03 21:48:19.596171 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:19.596003 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" event={"ID":"8a894e07-8a02-471c-90f6-0dc4db7c25e9","Type":"ContainerStarted","Data":"f2b5dc65b57cbb8d65fdd2441fdb387a2aabdc6900624ae3b52da2667269246a"} Nov 03 21:48:19.646198 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:19.646134 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" podStartSLOduration=1.6461159589999999 podStartE2EDuration="1.646115959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:19.638832624 +0000 UTC m=+1115.146060927" watchObservedRunningTime="2026-11-03 21:48:19.646115959 +0000 UTC m=+1115.153344239" Nov 03 21:48:19.815124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:19.815095 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" Nov 03 21:48:19.877874 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:19.877782 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f05ce451-3614-4af6-88e8-8793cf631ea1-kube-api-access-r52lg\" (UniqueName: \"kubernetes.io/projected/f05ce451-3614-4af6-88e8-8793cf631ea1-kube-api-access-r52lg\") pod \"f05ce451-3614-4af6-88e8-8793cf631ea1\" (UID: \"f05ce451-3614-4af6-88e8-8793cf631ea1\") " Nov 03 21:48:19.878053 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:19.877963 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f05ce451-3614-4af6-88e8-8793cf631ea1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05ce451-3614-4af6-88e8-8793cf631ea1-plugins-config-volume\") pod \"f05ce451-3614-4af6-88e8-8793cf631ea1\" (UID: \"f05ce451-3614-4af6-88e8-8793cf631ea1\") " Nov 03 21:48:19.878353 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:19.878329 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f05ce451-3614-4af6-88e8-8793cf631ea1-plugins-config-volume" pod "f05ce451-3614-4af6-88e8-8793cf631ea1" (UID: "f05ce451-3614-4af6-88e8-8793cf631ea1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:19.878550 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:19.878536 5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05ce451-3614-4af6-88e8-8793cf631ea1-plugins-config-volume\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:19.880030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:19.880008 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05ce451-3614-4af6-88e8-8793cf631ea1-kube-api-access-r52lg" pod "f05ce451-3614-4af6-88e8-8793cf631ea1" (UID: "f05ce451-3614-4af6-88e8-8793cf631ea1"). InnerVolumeSpecName "kube-api-access-r52lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:19.980414 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:19.980374 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r52lg\" (UniqueName: \"kubernetes.io/projected/f05ce451-3614-4af6-88e8-8793cf631ea1-kube-api-access-r52lg\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:20.616591 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:20.616550 5041 generic.go:356] "Generic (PLEG): container finished" podID="d975a235-5acc-47f6-9fc1-da6131f27b1b" containerID="f4df0a13f0e35259e11caaee6d9e5369d876cee1ef22b5f7b7088d0cf278bdac" exitCode=0 Nov 03 21:48:20.621095 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:20.621055 5041 generic.go:356] "Generic (PLEG): container finished" podID="8a894e07-8a02-471c-90f6-0dc4db7c25e9" containerID="acb75c4939541f7360323c73831a67c3b88d7a8484d9f753bbbdf23012b1037e" exitCode=0 Nov 03 21:48:20.623066 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:20.623034 5041 generic.go:356] "Generic (PLEG): container finished" podID="f05ce451-3614-4af6-88e8-8793cf631ea1" containerID="777cab1a0a4603d3b57dbc3d194b833a766e06e25e54dcefefd054ae152d1276" exitCode=0 Nov 03 21:48:20.623245 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:20.623128 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" Nov 03 21:48:20.630518 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:20.630480 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651" path="/var/lib/kubelet/pods/a3fd0876-cfb0-4dd0-9e7f-0c3e49bfe651/volumes" Nov 03 21:48:20.631316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:20.631270 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z" event={"ID":"d975a235-5acc-47f6-9fc1-da6131f27b1b","Type":"ContainerDied","Data":"f4df0a13f0e35259e11caaee6d9e5369d876cee1ef22b5f7b7088d0cf278bdac"} Nov 03 21:48:20.631461 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:20.631332 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" event={"ID":"8a894e07-8a02-471c-90f6-0dc4db7c25e9","Type":"ContainerDied","Data":"acb75c4939541f7360323c73831a67c3b88d7a8484d9f753bbbdf23012b1037e"} Nov 03 21:48:20.631461 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:20.631360 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" event={"ID":"f05ce451-3614-4af6-88e8-8793cf631ea1","Type":"ContainerDied","Data":"777cab1a0a4603d3b57dbc3d194b833a766e06e25e54dcefefd054ae152d1276"} Nov 03 21:48:20.631461 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:20.631388 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8" event={"ID":"f05ce451-3614-4af6-88e8-8793cf631ea1","Type":"ContainerDied","Data":"93ac4ca2b0d4f7222e2549a12ef64cde3a837c29d9c0f7addcd3b7c0a3a12983"} Nov 03 21:48:20.631461 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:20.631417 5041 scope.go:122] "RemoveContainer" containerID="777cab1a0a4603d3b57dbc3d194b833a766e06e25e54dcefefd054ae152d1276" Nov 03 21:48:20.648504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:20.648477 5041 scope.go:122] "RemoveContainer" containerID="777cab1a0a4603d3b57dbc3d194b833a766e06e25e54dcefefd054ae152d1276" Nov 03 21:48:20.648909 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:48:20.648887 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777cab1a0a4603d3b57dbc3d194b833a766e06e25e54dcefefd054ae152d1276\": container with ID starting with 777cab1a0a4603d3b57dbc3d194b833a766e06e25e54dcefefd054ae152d1276 not found: ID does not exist" containerID="777cab1a0a4603d3b57dbc3d194b833a766e06e25e54dcefefd054ae152d1276" Nov 03 21:48:20.648979 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:20.648924 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777cab1a0a4603d3b57dbc3d194b833a766e06e25e54dcefefd054ae152d1276"} err="failed to get container status \"777cab1a0a4603d3b57dbc3d194b833a766e06e25e54dcefefd054ae152d1276\": rpc error: code = NotFound desc = could not find container \"777cab1a0a4603d3b57dbc3d194b833a766e06e25e54dcefefd054ae152d1276\": container with ID starting with 777cab1a0a4603d3b57dbc3d194b833a766e06e25e54dcefefd054ae152d1276 not found: ID does not exist" Nov 03 21:48:20.756885 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:20.756791 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8"] Nov 03 21:48:20.766519 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:20.766469 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0a6da1-748b6dd54-4vdn8"] Nov 03 21:48:21.288007 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.287932 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" containerID="cri-o://faacd85db05e91963902cae054c4a1efac3cbc137a8e09d05c861188f4be4c10" gracePeriod=45 Nov 03 21:48:21.298218 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.298181 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" containerID="cri-o://1313d2f7abd621f5c2fccb8cf0e1bf6f2c22dcb52d5a984db947e85fab36cfa3" gracePeriod=45 Nov 03 21:48:21.428510 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.428468 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n"] Nov 03 21:48:21.445338 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.444328 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0a6da1-zxf2n"] Nov 03 21:48:21.464516 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.463739 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": dial tcp 10.132.0.74:8000: connect: connection refused" Nov 03 21:48:21.769110 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.768753 5041 generic.go:356] "Generic (PLEG): container finished" podID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerID="faacd85db05e91963902cae054c4a1efac3cbc137a8e09d05c861188f4be4c10" exitCode=0 Nov 03 21:48:21.769110 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.769037 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" event={"ID":"afb7a890-c3ba-4d5f-bce1-713c58220d7d","Type":"ContainerDied","Data":"faacd85db05e91963902cae054c4a1efac3cbc137a8e09d05c861188f4be4c10"} Nov 03 21:48:21.774292 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.773816 5041 generic.go:356] "Generic (PLEG): container finished" podID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerID="1313d2f7abd621f5c2fccb8cf0e1bf6f2c22dcb52d5a984db947e85fab36cfa3" exitCode=0 Nov 03 21:48:21.774292 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.774106 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" event={"ID":"c00e0aa7-7eb9-473d-b499-c0c9d4715981","Type":"ContainerDied","Data":"1313d2f7abd621f5c2fccb8cf0e1bf6f2c22dcb52d5a984db947e85fab36cfa3"} Nov 03 21:48:21.835207 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.834752 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:48:21.906225 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.906116 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00e0aa7-7eb9-473d-b499-c0c9d4715981-kube-api-access-qncsx\" (UniqueName: \"kubernetes.io/projected/c00e0aa7-7eb9-473d-b499-c0c9d4715981-kube-api-access-qncsx\") pod \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " Nov 03 21:48:21.906400 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.906251 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-home\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-home\") pod \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " Nov 03 21:48:21.906501 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.906478 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-dshm\") pod \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " Nov 03 21:48:21.906648 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.906610 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-tmp-dir\") pod \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " Nov 03 21:48:21.906823 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.906765 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-model-cache\") pod \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " Nov 03 21:48:21.906895 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.906884 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00e0aa7-7eb9-473d-b499-c0c9d4715981-tls-certs\" (UniqueName: \"kubernetes.io/secret/c00e0aa7-7eb9-473d-b499-c0c9d4715981-tls-certs\") pod \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\" (UID: \"c00e0aa7-7eb9-473d-b499-c0c9d4715981\") " Nov 03 21:48:21.907196 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.907162 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-home" pod "c00e0aa7-7eb9-473d-b499-c0c9d4715981" (UID: "c00e0aa7-7eb9-473d-b499-c0c9d4715981"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:21.907593 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.907567 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-tmp-dir" pod "c00e0aa7-7eb9-473d-b499-c0c9d4715981" (UID: "c00e0aa7-7eb9-473d-b499-c0c9d4715981"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:21.907733 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.907638 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-model-cache" pod "c00e0aa7-7eb9-473d-b499-c0c9d4715981" (UID: "c00e0aa7-7eb9-473d-b499-c0c9d4715981"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:21.907921 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.907899 5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-home\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:21.907972 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.907950 5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-tmp-dir\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:21.907972 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.907967 5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-model-cache\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:21.926451 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.926392 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00e0aa7-7eb9-473d-b499-c0c9d4715981-kube-api-access-qncsx" pod "c00e0aa7-7eb9-473d-b499-c0c9d4715981" (UID: "c00e0aa7-7eb9-473d-b499-c0c9d4715981"). InnerVolumeSpecName "kube-api-access-qncsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:21.927832 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.927791 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00e0aa7-7eb9-473d-b499-c0c9d4715981-tls-certs" pod "c00e0aa7-7eb9-473d-b499-c0c9d4715981" (UID: "c00e0aa7-7eb9-473d-b499-c0c9d4715981"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:21.928730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.928702 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-dshm" pod "c00e0aa7-7eb9-473d-b499-c0c9d4715981" (UID: "c00e0aa7-7eb9-473d-b499-c0c9d4715981"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:21.976562 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:21.975301 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:48:22.010899 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.009441 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-model-cache\") pod \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " Nov 03 21:48:22.010899 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.009531 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afb7a890-c3ba-4d5f-bce1-713c58220d7d-tls-certs\" (UniqueName: \"kubernetes.io/secret/afb7a890-c3ba-4d5f-bce1-713c58220d7d-tls-certs\") pod \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " Nov 03 21:48:22.010899 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.009579 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb7a890-c3ba-4d5f-bce1-713c58220d7d-kube-api-access-4r9bg\" (UniqueName: \"kubernetes.io/projected/afb7a890-c3ba-4d5f-bce1-713c58220d7d-kube-api-access-4r9bg\") pod \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " Nov 03 21:48:22.010899 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.009745 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-dshm\") pod \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " Nov 03 21:48:22.010899 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.009850 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-home\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-home\") pod \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " Nov 03 21:48:22.010899 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.009897 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-tmp-dir\") pod \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\" (UID: \"afb7a890-c3ba-4d5f-bce1-713c58220d7d\") " Nov 03 21:48:22.010899 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.010586 5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c00e0aa7-7eb9-473d-b499-c0c9d4715981-dshm\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.010899 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.010609 5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00e0aa7-7eb9-473d-b499-c0c9d4715981-tls-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.010899 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.010625 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qncsx\" (UniqueName: \"kubernetes.io/projected/c00e0aa7-7eb9-473d-b499-c0c9d4715981-kube-api-access-qncsx\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.011500 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.011240 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-model-cache" pod "afb7a890-c3ba-4d5f-bce1-713c58220d7d" (UID: "afb7a890-c3ba-4d5f-bce1-713c58220d7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:22.012381 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.012313 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-home" pod "afb7a890-c3ba-4d5f-bce1-713c58220d7d" (UID: "afb7a890-c3ba-4d5f-bce1-713c58220d7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:22.012574 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.012546 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-tmp-dir" pod "afb7a890-c3ba-4d5f-bce1-713c58220d7d" (UID: "afb7a890-c3ba-4d5f-bce1-713c58220d7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:22.020231 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.020174 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb7a890-c3ba-4d5f-bce1-713c58220d7d-kube-api-access-4r9bg" pod "afb7a890-c3ba-4d5f-bce1-713c58220d7d" (UID: "afb7a890-c3ba-4d5f-bce1-713c58220d7d"). InnerVolumeSpecName "kube-api-access-4r9bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:22.044089 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.044018 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afb7a890-c3ba-4d5f-bce1-713c58220d7d-tls-certs" pod "afb7a890-c3ba-4d5f-bce1-713c58220d7d" (UID: "afb7a890-c3ba-4d5f-bce1-713c58220d7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:22.047899 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.047828 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-dshm" pod "afb7a890-c3ba-4d5f-bce1-713c58220d7d" (UID: "afb7a890-c3ba-4d5f-bce1-713c58220d7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:22.112345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.112302 5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-home\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.112571 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.112553 5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-tmp-dir\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.112762 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.112691 5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-model-cache\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.112762 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.112709 5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb7a890-c3ba-4d5f-bce1-713c58220d7d-tls-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.112762 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.112724 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r9bg\" (UniqueName: \"kubernetes.io/projected/afb7a890-c3ba-4d5f-bce1-713c58220d7d-kube-api-access-4r9bg\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.112762 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.112740 5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb7a890-c3ba-4d5f-bce1-713c58220d7d-dshm\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.178258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.178222 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z" Nov 03 21:48:22.194523 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.194489 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" Nov 03 21:48:22.218654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.218610 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a894e07-8a02-471c-90f6-0dc4db7c25e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a894e07-8a02-471c-90f6-0dc4db7c25e9-maas-api-openshift-ca\") pod \"8a894e07-8a02-471c-90f6-0dc4db7c25e9\" (UID: \"8a894e07-8a02-471c-90f6-0dc4db7c25e9\") " Nov 03 21:48:22.218871 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.218678 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a894e07-8a02-471c-90f6-0dc4db7c25e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a894e07-8a02-471c-90f6-0dc4db7c25e9-maas-api-xks-ca\") pod \"8a894e07-8a02-471c-90f6-0dc4db7c25e9\" (UID: \"8a894e07-8a02-471c-90f6-0dc4db7c25e9\") " Nov 03 21:48:22.218871 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.218829 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d975a235-5acc-47f6-9fc1-da6131f27b1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d975a235-5acc-47f6-9fc1-da6131f27b1b-maas-api-openshift-ca\") pod \"d975a235-5acc-47f6-9fc1-da6131f27b1b\" (UID: \"d975a235-5acc-47f6-9fc1-da6131f27b1b\") " Nov 03 21:48:22.219099 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.218893 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d975a235-5acc-47f6-9fc1-da6131f27b1b-kube-api-access-qphgt\" (UniqueName: \"kubernetes.io/projected/d975a235-5acc-47f6-9fc1-da6131f27b1b-kube-api-access-qphgt\") pod \"d975a235-5acc-47f6-9fc1-da6131f27b1b\" (UID: \"d975a235-5acc-47f6-9fc1-da6131f27b1b\") " Nov 03 21:48:22.219099 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.219038 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a894e07-8a02-471c-90f6-0dc4db7c25e9-kube-api-access-rdxkp\" (UniqueName: \"kubernetes.io/projected/8a894e07-8a02-471c-90f6-0dc4db7c25e9-kube-api-access-rdxkp\") pod \"8a894e07-8a02-471c-90f6-0dc4db7c25e9\" (UID: \"8a894e07-8a02-471c-90f6-0dc4db7c25e9\") " Nov 03 21:48:22.219099 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.219076 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d975a235-5acc-47f6-9fc1-da6131f27b1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d975a235-5acc-47f6-9fc1-da6131f27b1b-maas-api-xks-ca\") pod \"d975a235-5acc-47f6-9fc1-da6131f27b1b\" (UID: \"d975a235-5acc-47f6-9fc1-da6131f27b1b\") " Nov 03 21:48:22.220988 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.220619 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a894e07-8a02-471c-90f6-0dc4db7c25e9-maas-api-openshift-ca" pod "8a894e07-8a02-471c-90f6-0dc4db7c25e9" (UID: "8a894e07-8a02-471c-90f6-0dc4db7c25e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:22.226318 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.226263 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d975a235-5acc-47f6-9fc1-da6131f27b1b-maas-api-openshift-ca" pod "d975a235-5acc-47f6-9fc1-da6131f27b1b" (UID: "d975a235-5acc-47f6-9fc1-da6131f27b1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:22.253812 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.253765 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a894e07-8a02-471c-90f6-0dc4db7c25e9-kube-api-access-rdxkp" pod "8a894e07-8a02-471c-90f6-0dc4db7c25e9" (UID: "8a894e07-8a02-471c-90f6-0dc4db7c25e9"). InnerVolumeSpecName "kube-api-access-rdxkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:22.254007 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.253846 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d975a235-5acc-47f6-9fc1-da6131f27b1b-kube-api-access-qphgt" pod "d975a235-5acc-47f6-9fc1-da6131f27b1b" (UID: "d975a235-5acc-47f6-9fc1-da6131f27b1b"). InnerVolumeSpecName "kube-api-access-qphgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:22.254234 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.254214 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d975a235-5acc-47f6-9fc1-da6131f27b1b-maas-api-xks-ca" pod "d975a235-5acc-47f6-9fc1-da6131f27b1b" (UID: "d975a235-5acc-47f6-9fc1-da6131f27b1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:22.255359 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.255321 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a894e07-8a02-471c-90f6-0dc4db7c25e9-maas-api-xks-ca" pod "8a894e07-8a02-471c-90f6-0dc4db7c25e9" (UID: "8a894e07-8a02-471c-90f6-0dc4db7c25e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:22.289385 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.289352 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" Nov 03 21:48:22.321168 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.321131 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabaa235-b28f-41a4-ad06-a10e60adc05e-kube-api-access-stg48\" (UniqueName: \"kubernetes.io/projected/cabaa235-b28f-41a4-ad06-a10e60adc05e-kube-api-access-stg48\") pod \"cabaa235-b28f-41a4-ad06-a10e60adc05e\" (UID: \"cabaa235-b28f-41a4-ad06-a10e60adc05e\") " Nov 03 21:48:22.321358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.321344 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabaa235-b28f-41a4-ad06-a10e60adc05e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabaa235-b28f-41a4-ad06-a10e60adc05e-plugins-config-volume\") pod \"cabaa235-b28f-41a4-ad06-a10e60adc05e\" (UID: \"cabaa235-b28f-41a4-ad06-a10e60adc05e\") " Nov 03 21:48:22.322282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.321863 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabaa235-b28f-41a4-ad06-a10e60adc05e-plugins-config-volume" pod "cabaa235-b28f-41a4-ad06-a10e60adc05e" (UID: "cabaa235-b28f-41a4-ad06-a10e60adc05e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:22.322282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.322092 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d975a235-5acc-47f6-9fc1-da6131f27b1b-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.322282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.322114 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qphgt\" (UniqueName: \"kubernetes.io/projected/d975a235-5acc-47f6-9fc1-da6131f27b1b-kube-api-access-qphgt\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.322282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.322129 5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabaa235-b28f-41a4-ad06-a10e60adc05e-plugins-config-volume\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.322282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.322142 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdxkp\" (UniqueName: \"kubernetes.io/projected/8a894e07-8a02-471c-90f6-0dc4db7c25e9-kube-api-access-rdxkp\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.322282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.322156 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d975a235-5acc-47f6-9fc1-da6131f27b1b-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.322282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.322171 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a894e07-8a02-471c-90f6-0dc4db7c25e9-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.322282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.322182 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a894e07-8a02-471c-90f6-0dc4db7c25e9-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.324214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.324180 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabaa235-b28f-41a4-ad06-a10e60adc05e-kube-api-access-stg48" pod "cabaa235-b28f-41a4-ad06-a10e60adc05e" (UID: "cabaa235-b28f-41a4-ad06-a10e60adc05e"). InnerVolumeSpecName "kube-api-access-stg48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:22.423630 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.423526 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stg48\" (UniqueName: \"kubernetes.io/projected/cabaa235-b28f-41a4-ad06-a10e60adc05e-kube-api-access-stg48\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:22.627956 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.627919 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2157382b-e7ba-44bf-9814-124cac52c539" path="/var/lib/kubelet/pods/2157382b-e7ba-44bf-9814-124cac52c539/volumes" Nov 03 21:48:22.628365 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.628349 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f05ce451-3614-4af6-88e8-8793cf631ea1" path="/var/lib/kubelet/pods/f05ce451-3614-4af6-88e8-8793cf631ea1/volumes" Nov 03 21:48:22.710129 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.710077 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:48:22.790042 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.790002 5041 generic.go:356] "Generic (PLEG): container finished" podID="cabaa235-b28f-41a4-ad06-a10e60adc05e" containerID="8702e5d74d46857bffc7c087b017643103bdd1e81d44ab751757d55c43f01d41" exitCode=0 Nov 03 21:48:22.790517 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.790090 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" Nov 03 21:48:22.790517 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.790286 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" event={"ID":"cabaa235-b28f-41a4-ad06-a10e60adc05e","Type":"ContainerDied","Data":"8702e5d74d46857bffc7c087b017643103bdd1e81d44ab751757d55c43f01d41"} Nov 03 21:48:22.790517 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.790337 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh" event={"ID":"cabaa235-b28f-41a4-ad06-a10e60adc05e","Type":"ContainerDied","Data":"710ec52de254eb27408f9c1ddffb671ebd83f75c0b6f6b9f30003458508b3f4e"} Nov 03 21:48:22.790517 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.790372 5041 scope.go:122] "RemoveContainer" containerID="8702e5d74d46857bffc7c087b017643103bdd1e81d44ab751757d55c43f01d41" Nov 03 21:48:22.793233 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.792934 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" Nov 03 21:48:22.793803 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.793740 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2" event={"ID":"afb7a890-c3ba-4d5f-bce1-713c58220d7d","Type":"ContainerDied","Data":"281cdfcb57783dae89fd78b787eeaa8747b2502b892fe86427a428a25adb6141"} Nov 03 21:48:22.796864 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.796837 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z" event={"ID":"d975a235-5acc-47f6-9fc1-da6131f27b1b","Type":"ContainerDied","Data":"4d089e0deee3dfa9435cf5530cdf7b271674c094d7263eb46d7c87ebe61d2683"} Nov 03 21:48:22.796980 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.796883 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d089e0deee3dfa9435cf5530cdf7b271674c094d7263eb46d7c87ebe61d2683" Nov 03 21:48:22.796980 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.796946 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z" Nov 03 21:48:22.800218 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.800196 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" Nov 03 21:48:22.800360 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.800238 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg" event={"ID":"8a894e07-8a02-471c-90f6-0dc4db7c25e9","Type":"ContainerDied","Data":"f2b5dc65b57cbb8d65fdd2441fdb387a2aabdc6900624ae3b52da2667269246a"} Nov 03 21:48:22.800360 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.800274 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2b5dc65b57cbb8d65fdd2441fdb387a2aabdc6900624ae3b52da2667269246a" Nov 03 21:48:22.803357 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.803327 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" event={"ID":"c00e0aa7-7eb9-473d-b499-c0c9d4715981","Type":"ContainerDied","Data":"b26ea0d43186511821e59c3764798b8c1bedcf0b3326c2e242ff76a9b3fd8837"} Nov 03 21:48:22.803499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.803385 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h" Nov 03 21:48:22.807768 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.807742 5041 scope.go:122] "RemoveContainer" containerID="8702e5d74d46857bffc7c087b017643103bdd1e81d44ab751757d55c43f01d41" Nov 03 21:48:22.808190 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:48:22.808160 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8702e5d74d46857bffc7c087b017643103bdd1e81d44ab751757d55c43f01d41\": container with ID starting with 8702e5d74d46857bffc7c087b017643103bdd1e81d44ab751757d55c43f01d41 not found: ID does not exist" containerID="8702e5d74d46857bffc7c087b017643103bdd1e81d44ab751757d55c43f01d41" Nov 03 21:48:22.808255 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.808203 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8702e5d74d46857bffc7c087b017643103bdd1e81d44ab751757d55c43f01d41"} err="failed to get container status \"8702e5d74d46857bffc7c087b017643103bdd1e81d44ab751757d55c43f01d41\": rpc error: code = NotFound desc = could not find container \"8702e5d74d46857bffc7c087b017643103bdd1e81d44ab751757d55c43f01d41\": container with ID starting with 8702e5d74d46857bffc7c087b017643103bdd1e81d44ab751757d55c43f01d41 not found: ID does not exist" Nov 03 21:48:22.808255 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.808227 5041 scope.go:122] "RemoveContainer" containerID="faacd85db05e91963902cae054c4a1efac3cbc137a8e09d05c861188f4be4c10" Nov 03 21:48:22.824953 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.824924 5041 scope.go:122] "RemoveContainer" containerID="1313d2f7abd621f5c2fccb8cf0e1bf6f2c22dcb52d5a984db947e85fab36cfa3" Nov 03 21:48:22.850336 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.850300 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh"] Nov 03 21:48:22.866004 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.865970 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4eced5-7547dd77cb-6nwvh"] Nov 03 21:48:22.922564 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.922516 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2"] Nov 03 21:48:22.930804 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.930758 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-27fdf4/distinct-w2-27fdf4-kserve-554dd588d4-cdws2"] Nov 03 21:48:22.978124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.978035 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h"] Nov 03 21:48:22.984357 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:22.984321 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-27fdf4/premium-w2-27fdf4-kserve-65bc566857-gbz7h"] Nov 03 21:48:24.624117 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:24.624081 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" path="/var/lib/kubelet/pods/afb7a890-c3ba-4d5f-bce1-713c58220d7d/volumes" Nov 03 21:48:24.625339 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:24.624781 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" path="/var/lib/kubelet/pods/c00e0aa7-7eb9-473d-b499-c0c9d4715981/volumes" Nov 03 21:48:24.625410 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:24.625351 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cabaa235-b28f-41a4-ad06-a10e60adc05e" path="/var/lib/kubelet/pods/cabaa235-b28f-41a4-ad06-a10e60adc05e/volumes" Nov 03 21:48:25.935562 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.935455 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj"] Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936482 5041 cpu_manager.go:414] "containerMap: removing container" podUID="d975a235-5acc-47f6-9fc1-da6131f27b1b" containerName="revoke-keys" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936509 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d975a235-5acc-47f6-9fc1-da6131f27b1b" containerName="revoke-keys" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936556 5041 cpu_manager.go:414] "containerMap: removing container" podUID="f05ce451-3614-4af6-88e8-8793cf631ea1" containerName="payload-processing" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936564 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f05ce451-3614-4af6-88e8-8793cf631ea1" containerName="payload-processing" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936584 5041 cpu_manager.go:414] "containerMap: removing container" podUID="8a894e07-8a02-471c-90f6-0dc4db7c25e9" containerName="revoke-keys" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936591 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a894e07-8a02-471c-90f6-0dc4db7c25e9" containerName="revoke-keys" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936617 5041 cpu_manager.go:414] "containerMap: removing container" podUID="cabaa235-b28f-41a4-ad06-a10e60adc05e" containerName="payload-processing" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936627 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabaa235-b28f-41a4-ad06-a10e60adc05e" containerName="payload-processing" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936639 5041 cpu_manager.go:414] "containerMap: removing container" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936646 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936682 5041 cpu_manager.go:414] "containerMap: removing container" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936689 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936925 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabaa235-b28f-41a4-ad06-a10e60adc05e" containerName="payload-processing" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936939 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb7a890-c3ba-4d5f-bce1-713c58220d7d" containerName="main" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936949 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00e0aa7-7eb9-473d-b499-c0c9d4715981" containerName="main" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936957 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d975a235-5acc-47f6-9fc1-da6131f27b1b" containerName="revoke-keys" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936967 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f05ce451-3614-4af6-88e8-8793cf631ea1" containerName="payload-processing" Nov 03 21:48:25.938911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.936975 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a894e07-8a02-471c-90f6-0dc4db7c25e9" containerName="revoke-keys" Nov 03 21:48:25.949772 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.944079 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:25.956377 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:25.955568 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj"] Nov 03 21:48:26.016205 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.016143 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istiod-ca-cert\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.016397 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.016237 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-envoy\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.016397 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.016303 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-workload-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.016397 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.016393 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-data\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.016605 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.016484 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-podinfo\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.016989 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.016961 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfchs\" (UniqueName: \"kubernetes.io/projected/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-kube-api-access-bfchs\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.017086 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.017036 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-token\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.017863 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.017833 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-credential-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.017973 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.017875 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-workload-certs\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.119255 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.119172 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-credential-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.119255 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.119248 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-workload-certs\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.119532 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.119286 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istiod-ca-cert\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.119532 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.119318 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-envoy\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.119532 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.119368 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-workload-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.119532 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.119436 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-data\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.119532 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.119510 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-podinfo\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.119806 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.119611 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfchs\" (UniqueName: \"kubernetes.io/projected/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-kube-api-access-bfchs\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.119806 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.119626 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-credential-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.119806 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.119652 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-token\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.120205 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.120156 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istiod-ca-cert\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.120315 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.120238 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-workload-socket\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.120462 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.120432 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-data\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.120536 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.120520 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-workload-certs\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.122655 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.122627 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-envoy\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.122940 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.122918 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-podinfo\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.141122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.141088 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-token\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.141389 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.141364 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfchs\" (UniqueName: \"kubernetes.io/projected/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-kube-api-access-bfchs\") pod \"e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.266325 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.266273 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:26.585745 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.582872 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb"] Nov 03 21:48:26.585745 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.583996 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" podUID="bc499793-421d-4550-af6d-691af2b07aa9" containerName="istio-proxy" containerID="cri-o://87f72ddcebb07a3581535f3bab87e30edbc284398a8b3bf852027963aff7a98d" gracePeriod=30 Nov 03 21:48:26.881400 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.881303 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg"] Nov 03 21:48:26.881400 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.881381 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2"] Nov 03 21:48:26.881628 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.881410 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l"] Nov 03 21:48:26.881628 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.881608 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" podUID="557fc6e5-610e-457d-895c-fa9746131887" containerName="istio-proxy" containerID="cri-o://f1e60ac9e83749d489d29f78235362e03bad2f1acbc4e53a44331fbec92a6f1b" gracePeriod=30 Nov 03 21:48:26.882123 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.882094 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" podUID="e9cb2586-4e9a-47cf-abd8-604c0f1264d5" containerName="istio-proxy" containerID="cri-o://2ec44d11eb69ed51556dc1bf695e17e85f744689e052fd8945b6dc82833f5f8e" gracePeriod=30 Nov 03 21:48:26.882324 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.882300 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" podUID="d87e1039-2359-4a69-a9ad-c029d9987a01" containerName="istio-proxy" containerID="cri-o://7b580bc43fc5670c4a0773cef014bd9513a9c2bd657465507d6a9e7415127217" gracePeriod=30 Nov 03 21:48:26.896332 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.896203 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm"] Nov 03 21:48:26.920287 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.919848 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-05bc27-stcmm"] Nov 03 21:48:26.989999 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.977332 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj"] Nov 03 21:48:26.999989 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:26.999944 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:48:27.000083 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:27.000046 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:48:27.000140 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:27.000097 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:48:27.971138 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:27.971098 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" event={"ID":"668e0b57-1f4e-441f-8d8b-fddc8524fcf6","Type":"ContainerStarted","Data":"6661c448bdef4714bdb8408c1ecdec3bdb28ee4dd65116a54092077b1f8d0de8"} Nov 03 21:48:27.971478 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:27.971440 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" event={"ID":"668e0b57-1f4e-441f-8d8b-fddc8524fcf6","Type":"ContainerStarted","Data":"79909632d901238ca6c9ab3f6ed7974bf751a561fffe66955217a0dee7e1a666"} Nov 03 21:48:28.030701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:28.029839 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" podStartSLOduration=3.029806535 podStartE2EDuration="3.029806535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:28.027476399 +0000 UTC m=+1123.534704680" watchObservedRunningTime="2026-11-03 21:48:28.029806535 +0000 UTC m=+1123.537034815" Nov 03 21:48:28.266623 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:28.266529 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:28.268938 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:28.268905 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 21:48:28.269081 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:28.268961 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" podUID="668e0b57-1f4e-441f-8d8b-fddc8524fcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 21:48:28.673565 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:28.673445 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d7e56bf-ba08-441e-925f-918461b36905" path="/var/lib/kubelet/pods/4d7e56bf-ba08-441e-925f-918461b36905/volumes" Nov 03 21:48:29.345302 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:29.330409 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 21:48:29.345302 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:29.330478 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" podUID="668e0b57-1f4e-441f-8d8b-fddc8524fcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 21:48:30.273689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:30.272963 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 21:48:30.273689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:30.273028 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" podUID="668e0b57-1f4e-441f-8d8b-fddc8524fcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 21:48:30.935570 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:30.933730 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6"] Nov 03 21:48:30.935570 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:30.935492 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" podUID="7d409dea-7691-4be3-9769-5e196850a3f2" containerName="istio-proxy" containerID="cri-o://46e44e5d19240a478c4a2a4b0e82b2d3b0f7f093805adfc431f8bf89705aa7d7" gracePeriod=30 Nov 03 21:48:31.332687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:31.332024 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 21:48:31.332687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:31.332108 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" podUID="668e0b57-1f4e-441f-8d8b-fddc8524fcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 21:48:31.345710 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:31.343338 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l"] Nov 03 21:48:31.345710 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:31.343640 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" podUID="72a848ed-b60e-4690-8933-8abdcac1d8ba" containerName="istio-proxy" containerID="cri-o://ffc32cf50a5f2f5ef2e5f0637b80b10ccb94d6118c77b1d4d48f766c26b2537a" gracePeriod=30 Nov 03 21:48:31.685143 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:31.683712 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj"] Nov 03 21:48:31.766018 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:31.744081 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" podUID="668e0b57-1f4e-441f-8d8b-fddc8524fcf6" containerName="istio-proxy" containerID="cri-o://6661c448bdef4714bdb8408c1ecdec3bdb28ee4dd65116a54092077b1f8d0de8" gracePeriod=30 Nov 03 21:48:32.053875 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:32.049752 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg"] Nov 03 21:48:32.053875 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:32.050041 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" podUID="ca8a2829-0a62-49af-b6ba-9546aa670350" containerName="istio-proxy" containerID="cri-o://95803f3ba7efe83abc13649fce4b12272b2ecee70ac4facc71dd570b1a33151c" gracePeriod=30 Nov 03 21:48:32.125796 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:32.124785 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm"] Nov 03 21:48:32.125796 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:32.125043 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" podUID="e29a22a0-dc41-4a86-ac9f-d6392e6c5472" containerName="istio-proxy" containerID="cri-o://18aebf2043dd1aed5e66119ec2b656a4f4d475acbc28e4fabbc675b3aafa73d5" gracePeriod=30 Nov 03 21:48:32.255770 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:32.255719 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr"] Nov 03 21:48:32.267693 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:32.256209 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" containerID="cri-o://ec0ccebca5738af3440a181f5aa49e75a6f81bcd757ee90a6a582d02360c2a67" gracePeriod=30 Nov 03 21:48:32.637763 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:32.616032 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 21:48:32.637763 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:32.616119 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" podUID="557fc6e5-610e-457d-895c-fa9746131887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 21:48:32.751686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:32.712119 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:48:32.751686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:32.712235 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" podUID="ca8a2829-0a62-49af-b6ba-9546aa670350" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:48:33.623238 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:33.623188 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 21:48:33.899836 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:33.892994 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:48:34.399815 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.399734 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:48:34.400063 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.400036 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" podUID="e9cb2586-4e9a-47cf-abd8-604c0f1264d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 21:48:34.673881 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.662140 5041 patch_prober.go:29] interesting pod/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:48:34.673881 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.662227 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" podUID="e29a22a0-dc41-4a86-ac9f-d6392e6c5472" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:48:34.724219 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.724186 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:48:34.725259 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:48:34.725192 5041 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.106s" Nov 03 21:48:34.725458 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.725442 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd"] Nov 03 21:48:34.728223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.728160 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" containerID="cri-o://711f15b5b8f73ec510f5dda14ead4b1df6b852205a3442ae8b0609cc761678ee" gracePeriod=30 Nov 03 21:48:34.755167 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.754920 5041 patch_prober.go:29] interesting pod/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:48:34.755167 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.754977 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" podUID="72a848ed-b60e-4690-8933-8abdcac1d8ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:48:34.761422 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.761288 5041 generic.go:356] "Generic (PLEG): container finished" podID="bc499793-421d-4550-af6d-691af2b07aa9" containerID="87f72ddcebb07a3581535f3bab87e30edbc284398a8b3bf852027963aff7a98d" exitCode=0 Nov 03 21:48:34.764776 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.764739 5041 generic.go:356] "Generic (PLEG): container finished" podID="e9cb2586-4e9a-47cf-abd8-604c0f1264d5" containerID="2ec44d11eb69ed51556dc1bf695e17e85f744689e052fd8945b6dc82833f5f8e" exitCode=0 Nov 03 21:48:34.767015 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.766830 5041 generic.go:356] "Generic (PLEG): container finished" podID="557fc6e5-610e-457d-895c-fa9746131887" containerID="f1e60ac9e83749d489d29f78235362e03bad2f1acbc4e53a44331fbec92a6f1b" exitCode=0 Nov 03 21:48:34.772109 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.772084 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg"] Nov 03 21:48:34.772436 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.772413 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" event={"ID":"bc499793-421d-4550-af6d-691af2b07aa9","Type":"ContainerDied","Data":"87f72ddcebb07a3581535f3bab87e30edbc284398a8b3bf852027963aff7a98d"} Nov 03 21:48:34.772631 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.772614 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4"] Nov 03 21:48:34.772779 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.772764 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" event={"ID":"e9cb2586-4e9a-47cf-abd8-604c0f1264d5","Type":"ContainerDied","Data":"2ec44d11eb69ed51556dc1bf695e17e85f744689e052fd8945b6dc82833f5f8e"} Nov 03 21:48:34.772893 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.772878 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" event={"ID":"557fc6e5-610e-457d-895c-fa9746131887","Type":"ContainerDied","Data":"f1e60ac9e83749d489d29f78235362e03bad2f1acbc4e53a44331fbec92a6f1b"} Nov 03 21:48:34.773158 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.773134 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" podUID="8ad94b74-be79-48fd-847d-491eb6d5943a" containerName="istio-proxy" containerID="cri-o://90984b1f19703f54a6c67c35ab35f4cf8060884219525e7400acc6779b4d02d3" gracePeriod=30 Nov 03 21:48:34.774085 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.774065 5041 scope.go:122] "RemoveContainer" containerID="87f72ddcebb07a3581535f3bab87e30edbc284398a8b3bf852027963aff7a98d" Nov 03 21:48:34.774264 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.772612 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" podUID="f738b002-3e77-4a00-9982-2cb525d164e6" containerName="istio-proxy" containerID="cri-o://85b11d0a2404d3902b7fda4df0d136d596953392540e87ec21dfd80040c158e5" gracePeriod=30 Nov 03 21:48:34.777244 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.777218 5041 generic.go:356] "Generic (PLEG): container finished" podID="d87e1039-2359-4a69-a9ad-c029d9987a01" containerID="7b580bc43fc5670c4a0773cef014bd9513a9c2bd657465507d6a9e7415127217" exitCode=0 Nov 03 21:48:34.777563 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.777544 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" event={"ID":"d87e1039-2359-4a69-a9ad-c029d9987a01","Type":"ContainerDied","Data":"7b580bc43fc5670c4a0773cef014bd9513a9c2bd657465507d6a9e7415127217"} Nov 03 21:48:34.777820 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.777805 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k"] Nov 03 21:48:34.778089 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.778069 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" podUID="523ee582-4304-412b-8269-cf9cf716c8a2" containerName="istio-proxy" containerID="cri-o://3b579db11de64b67ef6d213911848487c7d69d88904546ee67b0dce15fbc75a4" gracePeriod=30 Nov 03 21:48:34.930016 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.925180 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-credential-socket\") pod \"bc499793-421d-4550-af6d-691af2b07aa9\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " Nov 03 21:48:34.930016 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.925277 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc499793-421d-4550-af6d-691af2b07aa9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc499793-421d-4550-af6d-691af2b07aa9-istiod-ca-cert\") pod \"bc499793-421d-4550-af6d-691af2b07aa9\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " Nov 03 21:48:34.930016 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.925323 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc499793-421d-4550-af6d-691af2b07aa9-kube-api-access-pmxwf\" (UniqueName: \"kubernetes.io/projected/bc499793-421d-4550-af6d-691af2b07aa9-kube-api-access-pmxwf\") pod \"bc499793-421d-4550-af6d-691af2b07aa9\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " Nov 03 21:48:34.930016 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.925460 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-workload-socket\") pod \"bc499793-421d-4550-af6d-691af2b07aa9\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " Nov 03 21:48:34.930016 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.925512 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc499793-421d-4550-af6d-691af2b07aa9-istio-token\" (UniqueName: \"kubernetes.io/projected/bc499793-421d-4550-af6d-691af2b07aa9-istio-token\") pod \"bc499793-421d-4550-af6d-691af2b07aa9\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " Nov 03 21:48:34.930016 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.925594 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc499793-421d-4550-af6d-691af2b07aa9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc499793-421d-4550-af6d-691af2b07aa9-istio-podinfo\") pod \"bc499793-421d-4550-af6d-691af2b07aa9\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " Nov 03 21:48:34.930016 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.925701 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-workload-certs\") pod \"bc499793-421d-4550-af6d-691af2b07aa9\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " Nov 03 21:48:34.930016 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.925753 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-istio-data\") pod \"bc499793-421d-4550-af6d-691af2b07aa9\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " Nov 03 21:48:34.930016 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.925924 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-istio-envoy\") pod \"bc499793-421d-4550-af6d-691af2b07aa9\" (UID: \"bc499793-421d-4550-af6d-691af2b07aa9\") " Nov 03 21:48:34.948730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.945028 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:48:34.951825 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.951784 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-credential-socket" pod "bc499793-421d-4550-af6d-691af2b07aa9" (UID: "bc499793-421d-4550-af6d-691af2b07aa9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:34.959975 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.956932 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-istio-data" pod "bc499793-421d-4550-af6d-691af2b07aa9" (UID: "bc499793-421d-4550-af6d-691af2b07aa9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:34.959975 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.958494 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-workload-certs" pod "bc499793-421d-4550-af6d-691af2b07aa9" (UID: "bc499793-421d-4550-af6d-691af2b07aa9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:34.959975 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.959880 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc499793-421d-4550-af6d-691af2b07aa9-istiod-ca-cert" pod "bc499793-421d-4550-af6d-691af2b07aa9" (UID: "bc499793-421d-4550-af6d-691af2b07aa9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:34.959975 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.959944 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-workload-socket" pod "bc499793-421d-4550-af6d-691af2b07aa9" (UID: "bc499793-421d-4550-af6d-691af2b07aa9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:34.962309 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.962277 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-istio-envoy" pod "bc499793-421d-4550-af6d-691af2b07aa9" (UID: "bc499793-421d-4550-af6d-691af2b07aa9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:34.962769 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.962729 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc499793-421d-4550-af6d-691af2b07aa9-istio-podinfo" pod "bc499793-421d-4550-af6d-691af2b07aa9" (UID: "bc499793-421d-4550-af6d-691af2b07aa9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:34.963005 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.962902 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc499793-421d-4550-af6d-691af2b07aa9-istio-token" pod "bc499793-421d-4550-af6d-691af2b07aa9" (UID: "bc499793-421d-4550-af6d-691af2b07aa9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:34.963005 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.962974 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc499793-421d-4550-af6d-691af2b07aa9-kube-api-access-pmxwf" pod "bc499793-421d-4550-af6d-691af2b07aa9" (UID: "bc499793-421d-4550-af6d-691af2b07aa9"). InnerVolumeSpecName "kube-api-access-pmxwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:34.982916 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.982630 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:48:34.988683 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:34.988513 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:48:35.040689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.030152 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-credential-socket\") pod \"d87e1039-2359-4a69-a9ad-c029d9987a01\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " Nov 03 21:48:35.042378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.041115 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-workload-socket\") pod \"d87e1039-2359-4a69-a9ad-c029d9987a01\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " Nov 03 21:48:35.042378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.041208 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d87e1039-2359-4a69-a9ad-c029d9987a01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d87e1039-2359-4a69-a9ad-c029d9987a01-istiod-ca-cert\") pod \"d87e1039-2359-4a69-a9ad-c029d9987a01\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " Nov 03 21:48:35.042378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.041225 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-credential-socket" pod "d87e1039-2359-4a69-a9ad-c029d9987a01" (UID: "d87e1039-2359-4a69-a9ad-c029d9987a01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:35.042378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.041418 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87e1039-2359-4a69-a9ad-c029d9987a01-kube-api-access-tl657\" (UniqueName: \"kubernetes.io/projected/d87e1039-2359-4a69-a9ad-c029d9987a01-kube-api-access-tl657\") pod \"d87e1039-2359-4a69-a9ad-c029d9987a01\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " Nov 03 21:48:35.042378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.041457 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-workload-socket" pod "d87e1039-2359-4a69-a9ad-c029d9987a01" (UID: "d87e1039-2359-4a69-a9ad-c029d9987a01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:35.042378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.041477 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-data\") pod \"d87e1039-2359-4a69-a9ad-c029d9987a01\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " Nov 03 21:48:35.042378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.041538 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-workload-certs\") pod \"d87e1039-2359-4a69-a9ad-c029d9987a01\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " Nov 03 21:48:35.042378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.041580 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-token\" (UniqueName: \"kubernetes.io/projected/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-token\") pod \"d87e1039-2359-4a69-a9ad-c029d9987a01\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " Nov 03 21:48:35.042378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.041652 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-envoy\") pod \"d87e1039-2359-4a69-a9ad-c029d9987a01\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " Nov 03 21:48:35.042378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.041750 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-podinfo\") pod \"d87e1039-2359-4a69-a9ad-c029d9987a01\" (UID: \"d87e1039-2359-4a69-a9ad-c029d9987a01\") " Nov 03 21:48:35.042378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.041765 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87e1039-2359-4a69-a9ad-c029d9987a01-istiod-ca-cert" pod "d87e1039-2359-4a69-a9ad-c029d9987a01" (UID: "d87e1039-2359-4a69-a9ad-c029d9987a01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:35.042378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.041994 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-workload-certs" pod "d87e1039-2359-4a69-a9ad-c029d9987a01" (UID: "d87e1039-2359-4a69-a9ad-c029d9987a01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:35.043298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.042525 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc499793-421d-4550-af6d-691af2b07aa9-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.043298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.042548 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.043298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.042564 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.043298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.042578 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.043298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.042592 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.043298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.042605 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.043298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.042619 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc499793-421d-4550-af6d-691af2b07aa9-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.043298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.042633 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmxwf\" (UniqueName: \"kubernetes.io/projected/bc499793-421d-4550-af6d-691af2b07aa9-kube-api-access-pmxwf\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.043298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.042646 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.043298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.042676 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc499793-421d-4550-af6d-691af2b07aa9-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.043298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.042690 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.043298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.042705 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc499793-421d-4550-af6d-691af2b07aa9-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.043298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.042718 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d87e1039-2359-4a69-a9ad-c029d9987a01-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.043298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.043244 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-data" pod "d87e1039-2359-4a69-a9ad-c029d9987a01" (UID: "d87e1039-2359-4a69-a9ad-c029d9987a01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:35.045150 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.045112 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-podinfo" pod "d87e1039-2359-4a69-a9ad-c029d9987a01" (UID: "d87e1039-2359-4a69-a9ad-c029d9987a01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:35.046094 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.046065 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87e1039-2359-4a69-a9ad-c029d9987a01-kube-api-access-tl657" pod "d87e1039-2359-4a69-a9ad-c029d9987a01" (UID: "d87e1039-2359-4a69-a9ad-c029d9987a01"). InnerVolumeSpecName "kube-api-access-tl657". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:35.046219 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.046106 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-token" pod "d87e1039-2359-4a69-a9ad-c029d9987a01" (UID: "d87e1039-2359-4a69-a9ad-c029d9987a01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:35.047255 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.047231 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-envoy" pod "d87e1039-2359-4a69-a9ad-c029d9987a01" (UID: "d87e1039-2359-4a69-a9ad-c029d9987a01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:35.104007 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.103969 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:48:35.104176 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.104024 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" podUID="f738b002-3e77-4a00-9982-2cb525d164e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:48:35.144020 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.143983 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-istio-data\") pod \"557fc6e5-610e-457d-895c-fa9746131887\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " Nov 03 21:48:35.144217 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144045 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-token\" (UniqueName: \"kubernetes.io/projected/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-token\") pod \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " Nov 03 21:48:35.144217 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144124 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/557fc6e5-610e-457d-895c-fa9746131887-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/557fc6e5-610e-457d-895c-fa9746131887-istiod-ca-cert\") pod \"557fc6e5-610e-457d-895c-fa9746131887\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " Nov 03 21:48:35.144217 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144198 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-data\") pod \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " Nov 03 21:48:35.144394 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144226 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-workload-certs\") pod \"557fc6e5-610e-457d-895c-fa9746131887\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " Nov 03 21:48:35.144394 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144259 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-credential-socket\") pod \"557fc6e5-610e-457d-895c-fa9746131887\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " Nov 03 21:48:35.144394 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144300 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/557fc6e5-610e-457d-895c-fa9746131887-kube-api-access-cpl6j\" (UniqueName: \"kubernetes.io/projected/557fc6e5-610e-457d-895c-fa9746131887-kube-api-access-cpl6j\") pod \"557fc6e5-610e-457d-895c-fa9746131887\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " Nov 03 21:48:35.144394 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144303 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-istio-data" pod "557fc6e5-610e-457d-895c-fa9746131887" (UID: "557fc6e5-610e-457d-895c-fa9746131887"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:35.144394 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144390 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-podinfo\") pod \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " Nov 03 21:48:35.144648 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144429 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-workload-socket\") pod \"557fc6e5-610e-457d-895c-fa9746131887\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " Nov 03 21:48:35.144648 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144509 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-workload-certs\") pod \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " Nov 03 21:48:35.144648 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144516 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-data" pod "e9cb2586-4e9a-47cf-abd8-604c0f1264d5" (UID: "e9cb2586-4e9a-47cf-abd8-604c0f1264d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:35.144648 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144549 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/557fc6e5-610e-457d-895c-fa9746131887-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/557fc6e5-610e-457d-895c-fa9746131887-istio-podinfo\") pod \"557fc6e5-610e-457d-895c-fa9746131887\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " Nov 03 21:48:35.144648 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144629 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-envoy\") pod \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " Nov 03 21:48:35.144947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144675 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-istio-envoy\") pod \"557fc6e5-610e-457d-895c-fa9746131887\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " Nov 03 21:48:35.144947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144733 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-credential-socket" pod "557fc6e5-610e-457d-895c-fa9746131887" (UID: "557fc6e5-610e-457d-895c-fa9746131887"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:35.144947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144757 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-workload-socket\") pod \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " Nov 03 21:48:35.144947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144795 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-kube-api-access-hsljr\" (UniqueName: \"kubernetes.io/projected/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-kube-api-access-hsljr\") pod \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " Nov 03 21:48:35.144947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144816 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/557fc6e5-610e-457d-895c-fa9746131887-istiod-ca-cert" pod "557fc6e5-610e-457d-895c-fa9746131887" (UID: "557fc6e5-610e-457d-895c-fa9746131887"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:35.144947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144873 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istiod-ca-cert\") pod \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " Nov 03 21:48:35.144947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144916 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/557fc6e5-610e-457d-895c-fa9746131887-istio-token\" (UniqueName: \"kubernetes.io/projected/557fc6e5-610e-457d-895c-fa9746131887-istio-token\") pod \"557fc6e5-610e-457d-895c-fa9746131887\" (UID: \"557fc6e5-610e-457d-895c-fa9746131887\") " Nov 03 21:48:35.144947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144942 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-workload-certs" pod "e9cb2586-4e9a-47cf-abd8-604c0f1264d5" (UID: "e9cb2586-4e9a-47cf-abd8-604c0f1264d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:35.144947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144941 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-workload-certs" pod "557fc6e5-610e-457d-895c-fa9746131887" (UID: "557fc6e5-610e-457d-895c-fa9746131887"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:35.145357 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.144963 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-credential-socket\") pod \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\" (UID: \"e9cb2586-4e9a-47cf-abd8-604c0f1264d5\") " Nov 03 21:48:35.146122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.145634 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.146122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.145657 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.146122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.145684 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.146122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.145696 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d87e1039-2359-4a69-a9ad-c029d9987a01-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.146122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.145708 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.146122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.145721 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/557fc6e5-610e-457d-895c-fa9746131887-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.146122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.145733 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.146122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.145746 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.146122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.145759 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.146122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.145773 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.146122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.145788 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl657\" (UniqueName: \"kubernetes.io/projected/d87e1039-2359-4a69-a9ad-c029d9987a01-kube-api-access-tl657\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.146122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.146013 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-credential-socket" pod "e9cb2586-4e9a-47cf-abd8-604c0f1264d5" (UID: "e9cb2586-4e9a-47cf-abd8-604c0f1264d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:35.147040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.146874 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-token" pod "e9cb2586-4e9a-47cf-abd8-604c0f1264d5" (UID: "e9cb2586-4e9a-47cf-abd8-604c0f1264d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:35.147040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.147015 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-workload-socket" pod "557fc6e5-610e-457d-895c-fa9746131887" (UID: "557fc6e5-610e-457d-895c-fa9746131887"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:35.147346 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.147320 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-workload-socket" pod "e9cb2586-4e9a-47cf-abd8-604c0f1264d5" (UID: "e9cb2586-4e9a-47cf-abd8-604c0f1264d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:35.147456 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.147341 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-podinfo" pod "e9cb2586-4e9a-47cf-abd8-604c0f1264d5" (UID: "e9cb2586-4e9a-47cf-abd8-604c0f1264d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:35.147556 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.147460 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istiod-ca-cert" pod "e9cb2586-4e9a-47cf-abd8-604c0f1264d5" (UID: "e9cb2586-4e9a-47cf-abd8-604c0f1264d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:35.147651 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.147500 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557fc6e5-610e-457d-895c-fa9746131887-kube-api-access-cpl6j" pod "557fc6e5-610e-457d-895c-fa9746131887" (UID: "557fc6e5-610e-457d-895c-fa9746131887"). InnerVolumeSpecName "kube-api-access-cpl6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:35.149838 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.149802 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557fc6e5-610e-457d-895c-fa9746131887-istio-token" pod "557fc6e5-610e-457d-895c-fa9746131887" (UID: "557fc6e5-610e-457d-895c-fa9746131887"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:35.150019 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.149995 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/557fc6e5-610e-457d-895c-fa9746131887-istio-podinfo" pod "557fc6e5-610e-457d-895c-fa9746131887" (UID: "557fc6e5-610e-457d-895c-fa9746131887"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:35.150222 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.150199 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-envoy" pod "e9cb2586-4e9a-47cf-abd8-604c0f1264d5" (UID: "e9cb2586-4e9a-47cf-abd8-604c0f1264d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:35.150340 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.150305 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-kube-api-access-hsljr" pod "e9cb2586-4e9a-47cf-abd8-604c0f1264d5" (UID: "e9cb2586-4e9a-47cf-abd8-604c0f1264d5"). InnerVolumeSpecName "kube-api-access-hsljr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:35.151456 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.151434 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-istio-envoy" pod "557fc6e5-610e-457d-895c-fa9746131887" (UID: "557fc6e5-610e-457d-895c-fa9746131887"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:35.248371 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.248316 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.248560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.248397 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/557fc6e5-610e-457d-895c-fa9746131887-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.248560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.248429 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.248560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.248453 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.248560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.248469 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpl6j\" (UniqueName: \"kubernetes.io/projected/557fc6e5-610e-457d-895c-fa9746131887-kube-api-access-cpl6j\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.248560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.248483 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.248560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.248503 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.248560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.248520 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/557fc6e5-610e-457d-895c-fa9746131887-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.248560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.248533 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.248560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.248546 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/557fc6e5-610e-457d-895c-fa9746131887-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.248560 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.248564 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.249201 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.248579 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsljr\" (UniqueName: \"kubernetes.io/projected/e9cb2586-4e9a-47cf-abd8-604c0f1264d5-kube-api-access-hsljr\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:35.789696 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.789638 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" Nov 03 21:48:35.790170 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.789629 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" event={"ID":"d87e1039-2359-4a69-a9ad-c029d9987a01","Type":"ContainerDied","Data":"c4c630265a8f9cccc190e92aa3562bc85d13fcc3616e6b5547f6e8ac496ae433"} Nov 03 21:48:35.790170 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.789807 5041 scope.go:122] "RemoveContainer" containerID="7b580bc43fc5670c4a0773cef014bd9513a9c2bd657465507d6a9e7415127217" Nov 03 21:48:35.791656 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.791630 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" Nov 03 21:48:35.791656 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.791644 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb" event={"ID":"bc499793-421d-4550-af6d-691af2b07aa9","Type":"ContainerDied","Data":"5c169014dbbb7393b5b4dc8e6c8e3fb341a928501e7cdaff29850d629d6c7449"} Nov 03 21:48:35.793439 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.793398 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" event={"ID":"e9cb2586-4e9a-47cf-abd8-604c0f1264d5","Type":"ContainerDied","Data":"55f6fe46d3ec76d314820469f16cbad3f895d85550d941686a67b0333479861f"} Nov 03 21:48:35.793439 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.793428 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg" Nov 03 21:48:35.795634 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.795602 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" event={"ID":"557fc6e5-610e-457d-895c-fa9746131887","Type":"ContainerDied","Data":"6e9eaf7d0467111f11e92df9d4cd2693c40d7d9e927cc284438f161b64250acd"} Nov 03 21:48:35.795760 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.795735 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l" Nov 03 21:48:35.835753 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.835106 5041 scope.go:122] "RemoveContainer" containerID="2ec44d11eb69ed51556dc1bf695e17e85f744689e052fd8945b6dc82833f5f8e" Nov 03 21:48:35.835753 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.835246 5041 patch_prober.go:29] interesting pod/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:48:35.835753 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.835292 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2" podUID="d87e1039-2359-4a69-a9ad-c029d9987a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:48:35.851182 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.851149 5041 scope.go:122] "RemoveContainer" containerID="f1e60ac9e83749d489d29f78235362e03bad2f1acbc4e53a44331fbec92a6f1b" Nov 03 21:48:35.941517 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.941470 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg"] Nov 03 21:48:35.948269 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.948235 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-kj8vg"] Nov 03 21:48:35.971845 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.971814 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb"] Nov 03 21:48:35.978125 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:35.978091 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-8sjxb"] Nov 03 21:48:36.040484 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:36.040388 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2"] Nov 03 21:48:36.046624 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:36.046589 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-78hm2"] Nov 03 21:48:36.072210 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:36.072176 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l"] Nov 03 21:48:36.078108 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:36.078074 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4eced5-openshift-default-75c5d85f9f-l5z5l"] Nov 03 21:48:36.644254 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:36.644216 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="557fc6e5-610e-457d-895c-fa9746131887" path="/var/lib/kubelet/pods/557fc6e5-610e-457d-895c-fa9746131887/volumes" Nov 03 21:48:36.644859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:36.644841 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc499793-421d-4550-af6d-691af2b07aa9" path="/var/lib/kubelet/pods/bc499793-421d-4550-af6d-691af2b07aa9/volumes" Nov 03 21:48:36.645307 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:36.645293 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87e1039-2359-4a69-a9ad-c029d9987a01" path="/var/lib/kubelet/pods/d87e1039-2359-4a69-a9ad-c029d9987a01/volumes" Nov 03 21:48:36.646069 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:36.646054 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9cb2586-4e9a-47cf-abd8-604c0f1264d5" path="/var/lib/kubelet/pods/e9cb2586-4e9a-47cf-abd8-604c0f1264d5/volumes" Nov 03 21:48:37.362718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.362629 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:48:37.382586 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.382552 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-data\") pod \"ca8a2829-0a62-49af-b6ba-9546aa670350\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " Nov 03 21:48:37.382586 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.382599 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-workload-socket\") pod \"ca8a2829-0a62-49af-b6ba-9546aa670350\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " Nov 03 21:48:37.382896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.382683 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-podinfo\") pod \"ca8a2829-0a62-49af-b6ba-9546aa670350\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " Nov 03 21:48:37.382896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.382730 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-envoy\") pod \"ca8a2829-0a62-49af-b6ba-9546aa670350\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " Nov 03 21:48:37.382896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.382764 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-credential-socket\") pod \"ca8a2829-0a62-49af-b6ba-9546aa670350\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " Nov 03 21:48:37.382896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.382805 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca8a2829-0a62-49af-b6ba-9546aa670350-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8a2829-0a62-49af-b6ba-9546aa670350-istiod-ca-cert\") pod \"ca8a2829-0a62-49af-b6ba-9546aa670350\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " Nov 03 21:48:37.382896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.382837 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-token\" (UniqueName: \"kubernetes.io/projected/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-token\") pod \"ca8a2829-0a62-49af-b6ba-9546aa670350\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " Nov 03 21:48:37.382896 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.382876 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8a2829-0a62-49af-b6ba-9546aa670350-kube-api-access-wjlch\" (UniqueName: \"kubernetes.io/projected/ca8a2829-0a62-49af-b6ba-9546aa670350-kube-api-access-wjlch\") pod \"ca8a2829-0a62-49af-b6ba-9546aa670350\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " Nov 03 21:48:37.383611 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.383062 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-workload-certs\") pod \"ca8a2829-0a62-49af-b6ba-9546aa670350\" (UID: \"ca8a2829-0a62-49af-b6ba-9546aa670350\") " Nov 03 21:48:37.383798 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.383723 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-workload-socket" pod "ca8a2829-0a62-49af-b6ba-9546aa670350" (UID: "ca8a2829-0a62-49af-b6ba-9546aa670350"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:37.383901 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.383827 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-workload-certs" pod "ca8a2829-0a62-49af-b6ba-9546aa670350" (UID: "ca8a2829-0a62-49af-b6ba-9546aa670350"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:37.383981 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.383962 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-credential-socket" pod "ca8a2829-0a62-49af-b6ba-9546aa670350" (UID: "ca8a2829-0a62-49af-b6ba-9546aa670350"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:37.384778 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.384753 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-data" pod "ca8a2829-0a62-49af-b6ba-9546aa670350" (UID: "ca8a2829-0a62-49af-b6ba-9546aa670350"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:37.385374 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.385294 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca8a2829-0a62-49af-b6ba-9546aa670350-istiod-ca-cert" pod "ca8a2829-0a62-49af-b6ba-9546aa670350" (UID: "ca8a2829-0a62-49af-b6ba-9546aa670350"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:37.386990 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.386954 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-podinfo" pod "ca8a2829-0a62-49af-b6ba-9546aa670350" (UID: "ca8a2829-0a62-49af-b6ba-9546aa670350"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:37.386990 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.386983 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-token" pod "ca8a2829-0a62-49af-b6ba-9546aa670350" (UID: "ca8a2829-0a62-49af-b6ba-9546aa670350"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:37.387482 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.387264 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-envoy" pod "ca8a2829-0a62-49af-b6ba-9546aa670350" (UID: "ca8a2829-0a62-49af-b6ba-9546aa670350"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:37.387787 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.387764 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8a2829-0a62-49af-b6ba-9546aa670350-kube-api-access-wjlch" pod "ca8a2829-0a62-49af-b6ba-9546aa670350" (UID: "ca8a2829-0a62-49af-b6ba-9546aa670350"). InnerVolumeSpecName "kube-api-access-wjlch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:37.484800 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.484761 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.484800 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.484800 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.485054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.484814 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.485054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.484828 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.485054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.484843 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.485054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.484859 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca8a2829-0a62-49af-b6ba-9546aa670350-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.485054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.484872 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca8a2829-0a62-49af-b6ba-9546aa670350-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.485054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.484886 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjlch\" (UniqueName: \"kubernetes.io/projected/ca8a2829-0a62-49af-b6ba-9546aa670350-kube-api-access-wjlch\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.485054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.484899 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca8a2829-0a62-49af-b6ba-9546aa670350-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.523256 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.523225 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:48:37.588567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.586414 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-workload-socket\") pod \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " Nov 03 21:48:37.588567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.586498 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-podinfo\") pod \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " Nov 03 21:48:37.588567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.586597 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-data\") pod \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " Nov 03 21:48:37.588567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.586712 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-envoy\") pod \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " Nov 03 21:48:37.588567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.586781 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-token\" (UniqueName: \"kubernetes.io/projected/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-token\") pod \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " Nov 03 21:48:37.588567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.586842 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istiod-ca-cert\") pod \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " Nov 03 21:48:37.588567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.586878 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-credential-socket\") pod \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " Nov 03 21:48:37.588567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.586953 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-kube-api-access-2lf2t\" (UniqueName: \"kubernetes.io/projected/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-kube-api-access-2lf2t\") pod \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " Nov 03 21:48:37.588567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.586990 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-workload-certs\") pod \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\" (UID: \"e29a22a0-dc41-4a86-ac9f-d6392e6c5472\") " Nov 03 21:48:37.588567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.587342 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-workload-socket" pod "e29a22a0-dc41-4a86-ac9f-d6392e6c5472" (UID: "e29a22a0-dc41-4a86-ac9f-d6392e6c5472"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:37.588567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.587537 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-credential-socket" pod "e29a22a0-dc41-4a86-ac9f-d6392e6c5472" (UID: "e29a22a0-dc41-4a86-ac9f-d6392e6c5472"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:37.588567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.587674 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.588567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.587697 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.588567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.588190 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istiod-ca-cert" pod "e29a22a0-dc41-4a86-ac9f-d6392e6c5472" (UID: "e29a22a0-dc41-4a86-ac9f-d6392e6c5472"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:37.588567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.588554 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-workload-certs" pod "e29a22a0-dc41-4a86-ac9f-d6392e6c5472" (UID: "e29a22a0-dc41-4a86-ac9f-d6392e6c5472"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:37.589587 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.588616 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-data" pod "e29a22a0-dc41-4a86-ac9f-d6392e6c5472" (UID: "e29a22a0-dc41-4a86-ac9f-d6392e6c5472"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:37.591094 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.591057 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-podinfo" pod "e29a22a0-dc41-4a86-ac9f-d6392e6c5472" (UID: "e29a22a0-dc41-4a86-ac9f-d6392e6c5472"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:37.591225 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.591091 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-token" pod "e29a22a0-dc41-4a86-ac9f-d6392e6c5472" (UID: "e29a22a0-dc41-4a86-ac9f-d6392e6c5472"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:37.592596 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.592475 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-kube-api-access-2lf2t" pod "e29a22a0-dc41-4a86-ac9f-d6392e6c5472" (UID: "e29a22a0-dc41-4a86-ac9f-d6392e6c5472"). InnerVolumeSpecName "kube-api-access-2lf2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:37.594101 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.594007 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-envoy" pod "e29a22a0-dc41-4a86-ac9f-d6392e6c5472" (UID: "e29a22a0-dc41-4a86-ac9f-d6392e6c5472"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:37.689472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.689394 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.689472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.689426 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lf2t\" (UniqueName: \"kubernetes.io/projected/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-kube-api-access-2lf2t\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.689472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.689439 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.689472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.689447 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.689472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.689455 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.689472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.689463 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.689472 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.689471 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29a22a0-dc41-4a86-ac9f-d6392e6c5472-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.706562 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.706524 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:48:37.790218 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.790181 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-credential-socket\") pod \"72a848ed-b60e-4690-8933-8abdcac1d8ba\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " Nov 03 21:48:37.790394 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.790235 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a848ed-b60e-4690-8933-8abdcac1d8ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a848ed-b60e-4690-8933-8abdcac1d8ba-istiod-ca-cert\") pod \"72a848ed-b60e-4690-8933-8abdcac1d8ba\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " Nov 03 21:48:37.790394 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.790254 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-workload-socket\") pod \"72a848ed-b60e-4690-8933-8abdcac1d8ba\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " Nov 03 21:48:37.790394 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.790291 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-podinfo\") pod \"72a848ed-b60e-4690-8933-8abdcac1d8ba\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " Nov 03 21:48:37.790394 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.790349 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-data\") pod \"72a848ed-b60e-4690-8933-8abdcac1d8ba\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " Nov 03 21:48:37.790655 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.790410 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-token\" (UniqueName: \"kubernetes.io/projected/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-token\") pod \"72a848ed-b60e-4690-8933-8abdcac1d8ba\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " Nov 03 21:48:37.790655 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.790497 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a848ed-b60e-4690-8933-8abdcac1d8ba-kube-api-access-pj8br\" (UniqueName: \"kubernetes.io/projected/72a848ed-b60e-4690-8933-8abdcac1d8ba-kube-api-access-pj8br\") pod \"72a848ed-b60e-4690-8933-8abdcac1d8ba\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " Nov 03 21:48:37.790655 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.790524 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-workload-certs\") pod \"72a848ed-b60e-4690-8933-8abdcac1d8ba\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " Nov 03 21:48:37.790655 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.790552 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-envoy\") pod \"72a848ed-b60e-4690-8933-8abdcac1d8ba\" (UID: \"72a848ed-b60e-4690-8933-8abdcac1d8ba\") " Nov 03 21:48:37.791734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.791703 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a848ed-b60e-4690-8933-8abdcac1d8ba-istiod-ca-cert" pod "72a848ed-b60e-4690-8933-8abdcac1d8ba" (UID: "72a848ed-b60e-4690-8933-8abdcac1d8ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:37.791981 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.791906 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-workload-socket" pod "72a848ed-b60e-4690-8933-8abdcac1d8ba" (UID: "72a848ed-b60e-4690-8933-8abdcac1d8ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:37.792059 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.791980 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-credential-socket" pod "72a848ed-b60e-4690-8933-8abdcac1d8ba" (UID: "72a848ed-b60e-4690-8933-8abdcac1d8ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:37.792542 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.792470 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-workload-certs" pod "72a848ed-b60e-4690-8933-8abdcac1d8ba" (UID: "72a848ed-b60e-4690-8933-8abdcac1d8ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:37.792890 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.792870 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-data" pod "72a848ed-b60e-4690-8933-8abdcac1d8ba" (UID: "72a848ed-b60e-4690-8933-8abdcac1d8ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:37.794605 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.794584 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-envoy" pod "72a848ed-b60e-4690-8933-8abdcac1d8ba" (UID: "72a848ed-b60e-4690-8933-8abdcac1d8ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:37.795015 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.794987 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-podinfo" pod "72a848ed-b60e-4690-8933-8abdcac1d8ba" (UID: "72a848ed-b60e-4690-8933-8abdcac1d8ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:37.796544 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.796506 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a848ed-b60e-4690-8933-8abdcac1d8ba-kube-api-access-pj8br" pod "72a848ed-b60e-4690-8933-8abdcac1d8ba" (UID: "72a848ed-b60e-4690-8933-8abdcac1d8ba"). InnerVolumeSpecName "kube-api-access-pj8br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:37.798572 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.798535 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-token" pod "72a848ed-b60e-4690-8933-8abdcac1d8ba" (UID: "72a848ed-b60e-4690-8933-8abdcac1d8ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:37.834580 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.834542 5041 generic.go:356] "Generic (PLEG): container finished" podID="ca8a2829-0a62-49af-b6ba-9546aa670350" containerID="95803f3ba7efe83abc13649fce4b12272b2ecee70ac4facc71dd570b1a33151c" exitCode=0 Nov 03 21:48:37.834774 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.834616 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" Nov 03 21:48:37.834774 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.834631 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" event={"ID":"ca8a2829-0a62-49af-b6ba-9546aa670350","Type":"ContainerDied","Data":"95803f3ba7efe83abc13649fce4b12272b2ecee70ac4facc71dd570b1a33151c"} Nov 03 21:48:37.834774 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.834713 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg" event={"ID":"ca8a2829-0a62-49af-b6ba-9546aa670350","Type":"ContainerDied","Data":"f9fbfc4d138d22741bcf8d880e728fdc99bcce818ec1a909923e4b685cebf7f2"} Nov 03 21:48:37.834774 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.834752 5041 scope.go:122] "RemoveContainer" containerID="95803f3ba7efe83abc13649fce4b12272b2ecee70ac4facc71dd570b1a33151c" Nov 03 21:48:37.836735 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.836708 5041 generic.go:356] "Generic (PLEG): container finished" podID="72a848ed-b60e-4690-8933-8abdcac1d8ba" containerID="ffc32cf50a5f2f5ef2e5f0637b80b10ccb94d6118c77b1d4d48f766c26b2537a" exitCode=0 Nov 03 21:48:37.837090 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.837059 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" event={"ID":"72a848ed-b60e-4690-8933-8abdcac1d8ba","Type":"ContainerDied","Data":"ffc32cf50a5f2f5ef2e5f0637b80b10ccb94d6118c77b1d4d48f766c26b2537a"} Nov 03 21:48:37.837221 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.837126 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" event={"ID":"72a848ed-b60e-4690-8933-8abdcac1d8ba","Type":"ContainerDied","Data":"4ec065c96a29762e97443864f8d94d201a174da9ee735e3a5e2ee57ec53e7b8b"} Nov 03 21:48:37.837221 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.837206 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l" Nov 03 21:48:37.842944 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.842902 5041 generic.go:356] "Generic (PLEG): container finished" podID="e29a22a0-dc41-4a86-ac9f-d6392e6c5472" containerID="18aebf2043dd1aed5e66119ec2b656a4f4d475acbc28e4fabbc675b3aafa73d5" exitCode=0 Nov 03 21:48:37.843198 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.843175 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" event={"ID":"e29a22a0-dc41-4a86-ac9f-d6392e6c5472","Type":"ContainerDied","Data":"18aebf2043dd1aed5e66119ec2b656a4f4d475acbc28e4fabbc675b3aafa73d5"} Nov 03 21:48:37.843281 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.843244 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" event={"ID":"e29a22a0-dc41-4a86-ac9f-d6392e6c5472","Type":"ContainerDied","Data":"961a0747646ad75f831c49801a1c46453f1b90c2075708b397fc12c478f7d506"} Nov 03 21:48:37.843281 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.843258 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm" Nov 03 21:48:37.851150 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.851029 5041 scope.go:122] "RemoveContainer" containerID="95803f3ba7efe83abc13649fce4b12272b2ecee70ac4facc71dd570b1a33151c" Nov 03 21:48:37.852357 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:48:37.852326 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95803f3ba7efe83abc13649fce4b12272b2ecee70ac4facc71dd570b1a33151c\": container with ID starting with 95803f3ba7efe83abc13649fce4b12272b2ecee70ac4facc71dd570b1a33151c not found: ID does not exist" containerID="95803f3ba7efe83abc13649fce4b12272b2ecee70ac4facc71dd570b1a33151c" Nov 03 21:48:37.852805 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.852769 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95803f3ba7efe83abc13649fce4b12272b2ecee70ac4facc71dd570b1a33151c"} err="failed to get container status \"95803f3ba7efe83abc13649fce4b12272b2ecee70ac4facc71dd570b1a33151c\": rpc error: code = NotFound desc = could not find container \"95803f3ba7efe83abc13649fce4b12272b2ecee70ac4facc71dd570b1a33151c\": container with ID starting with 95803f3ba7efe83abc13649fce4b12272b2ecee70ac4facc71dd570b1a33151c not found: ID does not exist" Nov 03 21:48:37.852906 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.852807 5041 scope.go:122] "RemoveContainer" containerID="ffc32cf50a5f2f5ef2e5f0637b80b10ccb94d6118c77b1d4d48f766c26b2537a" Nov 03 21:48:37.870738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.870655 5041 scope.go:122] "RemoveContainer" containerID="ffc32cf50a5f2f5ef2e5f0637b80b10ccb94d6118c77b1d4d48f766c26b2537a" Nov 03 21:48:37.871067 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:48:37.871048 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffc32cf50a5f2f5ef2e5f0637b80b10ccb94d6118c77b1d4d48f766c26b2537a\": container with ID starting with ffc32cf50a5f2f5ef2e5f0637b80b10ccb94d6118c77b1d4d48f766c26b2537a not found: ID does not exist" containerID="ffc32cf50a5f2f5ef2e5f0637b80b10ccb94d6118c77b1d4d48f766c26b2537a" Nov 03 21:48:37.871147 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.871075 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffc32cf50a5f2f5ef2e5f0637b80b10ccb94d6118c77b1d4d48f766c26b2537a"} err="failed to get container status \"ffc32cf50a5f2f5ef2e5f0637b80b10ccb94d6118c77b1d4d48f766c26b2537a\": rpc error: code = NotFound desc = could not find container \"ffc32cf50a5f2f5ef2e5f0637b80b10ccb94d6118c77b1d4d48f766c26b2537a\": container with ID starting with ffc32cf50a5f2f5ef2e5f0637b80b10ccb94d6118c77b1d4d48f766c26b2537a not found: ID does not exist" Nov 03 21:48:37.871147 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.871094 5041 scope.go:122] "RemoveContainer" containerID="18aebf2043dd1aed5e66119ec2b656a4f4d475acbc28e4fabbc675b3aafa73d5" Nov 03 21:48:37.892972 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.892929 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.892972 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.892972 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj8br\" (UniqueName: \"kubernetes.io/projected/72a848ed-b60e-4690-8933-8abdcac1d8ba-kube-api-access-pj8br\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.893231 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.892989 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.893231 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.893005 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.893231 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.893030 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.893231 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.893043 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a848ed-b60e-4690-8933-8abdcac1d8ba-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.893231 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.893057 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.893231 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.893070 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.893231 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.893083 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a848ed-b60e-4690-8933-8abdcac1d8ba-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:37.893617 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.893576 5041 scope.go:122] "RemoveContainer" containerID="18aebf2043dd1aed5e66119ec2b656a4f4d475acbc28e4fabbc675b3aafa73d5" Nov 03 21:48:37.894578 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:48:37.894549 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18aebf2043dd1aed5e66119ec2b656a4f4d475acbc28e4fabbc675b3aafa73d5\": container with ID starting with 18aebf2043dd1aed5e66119ec2b656a4f4d475acbc28e4fabbc675b3aafa73d5 not found: ID does not exist" containerID="18aebf2043dd1aed5e66119ec2b656a4f4d475acbc28e4fabbc675b3aafa73d5" Nov 03 21:48:37.894712 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.894587 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18aebf2043dd1aed5e66119ec2b656a4f4d475acbc28e4fabbc675b3aafa73d5"} err="failed to get container status \"18aebf2043dd1aed5e66119ec2b656a4f4d475acbc28e4fabbc675b3aafa73d5\": rpc error: code = NotFound desc = could not find container \"18aebf2043dd1aed5e66119ec2b656a4f4d475acbc28e4fabbc675b3aafa73d5\": container with ID starting with 18aebf2043dd1aed5e66119ec2b656a4f4d475acbc28e4fabbc675b3aafa73d5 not found: ID does not exist" Nov 03 21:48:37.900018 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.899987 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg"] Nov 03 21:48:37.906194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.906163 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-v4jmg"] Nov 03 21:48:37.977701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.977631 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l"] Nov 03 21:48:37.997977 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:37.997701 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pjp7l"] Nov 03 21:48:38.177831 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.151954 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm"] Nov 03 21:48:38.215704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.191079 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-gwqdm"] Nov 03 21:48:38.606887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.606516 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:48:38.626535 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.626504 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a848ed-b60e-4690-8933-8abdcac1d8ba" path="/var/lib/kubelet/pods/72a848ed-b60e-4690-8933-8abdcac1d8ba/volumes" Nov 03 21:48:38.627109 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.627083 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca8a2829-0a62-49af-b6ba-9546aa670350" path="/var/lib/kubelet/pods/ca8a2829-0a62-49af-b6ba-9546aa670350/volumes" Nov 03 21:48:38.627709 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.627691 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e29a22a0-dc41-4a86-ac9f-d6392e6c5472" path="/var/lib/kubelet/pods/e29a22a0-dc41-4a86-ac9f-d6392e6c5472/volumes" Nov 03 21:48:38.630106 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.630085 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d409dea-7691-4be3-9769-5e196850a3f2-istio-token\" (UniqueName: \"kubernetes.io/projected/7d409dea-7691-4be3-9769-5e196850a3f2-istio-token\") pod \"7d409dea-7691-4be3-9769-5e196850a3f2\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " Nov 03 21:48:38.630215 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.630153 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-istio-data\") pod \"7d409dea-7691-4be3-9769-5e196850a3f2\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " Nov 03 21:48:38.630270 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.630233 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d409dea-7691-4be3-9769-5e196850a3f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d409dea-7691-4be3-9769-5e196850a3f2-istiod-ca-cert\") pod \"7d409dea-7691-4be3-9769-5e196850a3f2\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " Nov 03 21:48:38.630360 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.630341 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d409dea-7691-4be3-9769-5e196850a3f2-kube-api-access-vn4f4\" (UniqueName: \"kubernetes.io/projected/7d409dea-7691-4be3-9769-5e196850a3f2-kube-api-access-vn4f4\") pod \"7d409dea-7691-4be3-9769-5e196850a3f2\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " Nov 03 21:48:38.630447 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.630429 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-credential-socket\") pod \"7d409dea-7691-4be3-9769-5e196850a3f2\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " Nov 03 21:48:38.630546 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.630513 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-istio-data" pod "7d409dea-7691-4be3-9769-5e196850a3f2" (UID: "7d409dea-7691-4be3-9769-5e196850a3f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:38.630723 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.630629 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-credential-socket" pod "7d409dea-7691-4be3-9769-5e196850a3f2" (UID: "7d409dea-7691-4be3-9769-5e196850a3f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:38.630816 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.630791 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d409dea-7691-4be3-9769-5e196850a3f2-istiod-ca-cert" pod "7d409dea-7691-4be3-9769-5e196850a3f2" (UID: "7d409dea-7691-4be3-9769-5e196850a3f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:38.630938 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.630910 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-istio-envoy\") pod \"7d409dea-7691-4be3-9769-5e196850a3f2\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " Nov 03 21:48:38.631030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.630974 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-workload-certs\") pod \"7d409dea-7691-4be3-9769-5e196850a3f2\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " Nov 03 21:48:38.631102 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.631041 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d409dea-7691-4be3-9769-5e196850a3f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d409dea-7691-4be3-9769-5e196850a3f2-istio-podinfo\") pod \"7d409dea-7691-4be3-9769-5e196850a3f2\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " Nov 03 21:48:38.631102 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.631085 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-workload-socket\") pod \"7d409dea-7691-4be3-9769-5e196850a3f2\" (UID: \"7d409dea-7691-4be3-9769-5e196850a3f2\") " Nov 03 21:48:38.631650 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.631629 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.631767 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.631652 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d409dea-7691-4be3-9769-5e196850a3f2-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.631767 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.631684 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.632620 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.632594 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-workload-socket" pod "7d409dea-7691-4be3-9769-5e196850a3f2" (UID: "7d409dea-7691-4be3-9769-5e196850a3f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:38.632822 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.632620 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-workload-certs" pod "7d409dea-7691-4be3-9769-5e196850a3f2" (UID: "7d409dea-7691-4be3-9769-5e196850a3f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:38.633442 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.633150 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d409dea-7691-4be3-9769-5e196850a3f2-kube-api-access-vn4f4" pod "7d409dea-7691-4be3-9769-5e196850a3f2" (UID: "7d409dea-7691-4be3-9769-5e196850a3f2"). InnerVolumeSpecName "kube-api-access-vn4f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:38.633442 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.633390 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d409dea-7691-4be3-9769-5e196850a3f2-istio-token" pod "7d409dea-7691-4be3-9769-5e196850a3f2" (UID: "7d409dea-7691-4be3-9769-5e196850a3f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:38.634838 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.634815 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d409dea-7691-4be3-9769-5e196850a3f2-istio-podinfo" pod "7d409dea-7691-4be3-9769-5e196850a3f2" (UID: "7d409dea-7691-4be3-9769-5e196850a3f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:38.634916 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.634897 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-istio-envoy" pod "7d409dea-7691-4be3-9769-5e196850a3f2" (UID: "7d409dea-7691-4be3-9769-5e196850a3f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:38.652284 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.652250 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:48:38.733764 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.733723 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-workload-socket\") pod \"fcda3325-5e04-4611-9a1e-48075b67d03b\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " Nov 03 21:48:38.733930 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.733808 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-credential-socket\") pod \"fcda3325-5e04-4611-9a1e-48075b67d03b\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " Nov 03 21:48:38.733930 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.733895 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-data\") pod \"fcda3325-5e04-4611-9a1e-48075b67d03b\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " Nov 03 21:48:38.733930 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.733923 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-envoy\") pod \"fcda3325-5e04-4611-9a1e-48075b67d03b\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " Nov 03 21:48:38.734062 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.734014 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-podinfo\") pod \"fcda3325-5e04-4611-9a1e-48075b67d03b\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " Nov 03 21:48:38.734062 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.734053 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-token\" (UniqueName: \"kubernetes.io/projected/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-token\") pod \"fcda3325-5e04-4611-9a1e-48075b67d03b\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " Nov 03 21:48:38.734131 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.734095 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-workload-certs\") pod \"fcda3325-5e04-4611-9a1e-48075b67d03b\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " Nov 03 21:48:38.734220 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.734204 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcda3325-5e04-4611-9a1e-48075b67d03b-kube-api-access-96g8z\" (UniqueName: \"kubernetes.io/projected/fcda3325-5e04-4611-9a1e-48075b67d03b-kube-api-access-96g8z\") pod \"fcda3325-5e04-4611-9a1e-48075b67d03b\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " Nov 03 21:48:38.734319 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.734305 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcda3325-5e04-4611-9a1e-48075b67d03b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcda3325-5e04-4611-9a1e-48075b67d03b-istiod-ca-cert\") pod \"fcda3325-5e04-4611-9a1e-48075b67d03b\" (UID: \"fcda3325-5e04-4611-9a1e-48075b67d03b\") " Nov 03 21:48:38.734813 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.734790 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn4f4\" (UniqueName: \"kubernetes.io/projected/7d409dea-7691-4be3-9769-5e196850a3f2-kube-api-access-vn4f4\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.734922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.734818 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.734922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.734834 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.734922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.734847 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d409dea-7691-4be3-9769-5e196850a3f2-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.734922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.734861 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d409dea-7691-4be3-9769-5e196850a3f2-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.734922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.734875 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d409dea-7691-4be3-9769-5e196850a3f2-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.737604 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.737574 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcda3325-5e04-4611-9a1e-48075b67d03b-istiod-ca-cert" pod "fcda3325-5e04-4611-9a1e-48075b67d03b" (UID: "fcda3325-5e04-4611-9a1e-48075b67d03b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:38.738970 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.738941 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-workload-socket" pod "fcda3325-5e04-4611-9a1e-48075b67d03b" (UID: "fcda3325-5e04-4611-9a1e-48075b67d03b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:38.739198 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.739176 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-credential-socket" pod "fcda3325-5e04-4611-9a1e-48075b67d03b" (UID: "fcda3325-5e04-4611-9a1e-48075b67d03b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:38.739675 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.739620 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-data" pod "fcda3325-5e04-4611-9a1e-48075b67d03b" (UID: "fcda3325-5e04-4611-9a1e-48075b67d03b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:38.741121 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.741094 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-workload-certs" pod "fcda3325-5e04-4611-9a1e-48075b67d03b" (UID: "fcda3325-5e04-4611-9a1e-48075b67d03b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:38.741738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.741714 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-envoy" pod "fcda3325-5e04-4611-9a1e-48075b67d03b" (UID: "fcda3325-5e04-4611-9a1e-48075b67d03b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:38.744463 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.744425 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcda3325-5e04-4611-9a1e-48075b67d03b-kube-api-access-96g8z" pod "fcda3325-5e04-4611-9a1e-48075b67d03b" (UID: "fcda3325-5e04-4611-9a1e-48075b67d03b"). InnerVolumeSpecName "kube-api-access-96g8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:38.756405 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.756367 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-token" pod "fcda3325-5e04-4611-9a1e-48075b67d03b" (UID: "fcda3325-5e04-4611-9a1e-48075b67d03b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:38.756838 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.756800 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-podinfo" pod "fcda3325-5e04-4611-9a1e-48075b67d03b" (UID: "fcda3325-5e04-4611-9a1e-48075b67d03b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:38.837955 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.837891 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.838167 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.838146 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.838256 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.838171 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.838256 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.838186 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.838256 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.838198 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.838256 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.838210 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcda3325-5e04-4611-9a1e-48075b67d03b-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.838256 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.838228 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcda3325-5e04-4611-9a1e-48075b67d03b-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.838256 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.838242 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96g8z\" (UniqueName: \"kubernetes.io/projected/fcda3325-5e04-4611-9a1e-48075b67d03b-kube-api-access-96g8z\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.838256 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.838255 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcda3325-5e04-4611-9a1e-48075b67d03b-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:38.857495 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.857451 5041 generic.go:356] "Generic (PLEG): container finished" podID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerID="ec0ccebca5738af3440a181f5aa49e75a6f81bcd757ee90a6a582d02360c2a67" exitCode=0 Nov 03 21:48:38.857690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.857506 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" Nov 03 21:48:38.857690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.857540 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" event={"ID":"fcda3325-5e04-4611-9a1e-48075b67d03b","Type":"ContainerDied","Data":"ec0ccebca5738af3440a181f5aa49e75a6f81bcd757ee90a6a582d02360c2a67"} Nov 03 21:48:38.858450 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.858417 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr" event={"ID":"fcda3325-5e04-4611-9a1e-48075b67d03b","Type":"ContainerDied","Data":"50a47dbab70c3880188b0738e5b5e40a39a0ecc51ea00d44136079be31febd46"} Nov 03 21:48:38.858605 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.858482 5041 scope.go:122] "RemoveContainer" containerID="ec0ccebca5738af3440a181f5aa49e75a6f81bcd757ee90a6a582d02360c2a67" Nov 03 21:48:38.917111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.917064 5041 generic.go:356] "Generic (PLEG): container finished" podID="7d409dea-7691-4be3-9769-5e196850a3f2" containerID="46e44e5d19240a478c4a2a4b0e82b2d3b0f7f093805adfc431f8bf89705aa7d7" exitCode=0 Nov 03 21:48:38.917387 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.917369 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" event={"ID":"7d409dea-7691-4be3-9769-5e196850a3f2","Type":"ContainerDied","Data":"46e44e5d19240a478c4a2a4b0e82b2d3b0f7f093805adfc431f8bf89705aa7d7"} Nov 03 21:48:38.917681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.917641 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" Nov 03 21:48:38.917922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.917904 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6" event={"ID":"7d409dea-7691-4be3-9769-5e196850a3f2","Type":"ContainerDied","Data":"4107fbc8533ca0ad0841c241866bdad83a4b69a37df8157ba6a90bfe51866034"} Nov 03 21:48:38.920618 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.920596 5041 scope.go:122] "RemoveContainer" containerID="ec0ccebca5738af3440a181f5aa49e75a6f81bcd757ee90a6a582d02360c2a67" Nov 03 21:48:38.921757 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:48:38.921725 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0ccebca5738af3440a181f5aa49e75a6f81bcd757ee90a6a582d02360c2a67\": container with ID starting with ec0ccebca5738af3440a181f5aa49e75a6f81bcd757ee90a6a582d02360c2a67 not found: ID does not exist" containerID="ec0ccebca5738af3440a181f5aa49e75a6f81bcd757ee90a6a582d02360c2a67" Nov 03 21:48:38.921915 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.921889 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0ccebca5738af3440a181f5aa49e75a6f81bcd757ee90a6a582d02360c2a67"} err="failed to get container status \"ec0ccebca5738af3440a181f5aa49e75a6f81bcd757ee90a6a582d02360c2a67\": rpc error: code = NotFound desc = could not find container \"ec0ccebca5738af3440a181f5aa49e75a6f81bcd757ee90a6a582d02360c2a67\": container with ID starting with ec0ccebca5738af3440a181f5aa49e75a6f81bcd757ee90a6a582d02360c2a67 not found: ID does not exist" Nov 03 21:48:38.922009 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.921997 5041 scope.go:122] "RemoveContainer" containerID="46e44e5d19240a478c4a2a4b0e82b2d3b0f7f093805adfc431f8bf89705aa7d7" Nov 03 21:48:38.937031 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.937000 5041 scope.go:122] "RemoveContainer" containerID="46e44e5d19240a478c4a2a4b0e82b2d3b0f7f093805adfc431f8bf89705aa7d7" Nov 03 21:48:38.937437 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:48:38.937411 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e44e5d19240a478c4a2a4b0e82b2d3b0f7f093805adfc431f8bf89705aa7d7\": container with ID starting with 46e44e5d19240a478c4a2a4b0e82b2d3b0f7f093805adfc431f8bf89705aa7d7 not found: ID does not exist" containerID="46e44e5d19240a478c4a2a4b0e82b2d3b0f7f093805adfc431f8bf89705aa7d7" Nov 03 21:48:38.937502 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.937450 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e44e5d19240a478c4a2a4b0e82b2d3b0f7f093805adfc431f8bf89705aa7d7"} err="failed to get container status \"46e44e5d19240a478c4a2a4b0e82b2d3b0f7f093805adfc431f8bf89705aa7d7\": rpc error: code = NotFound desc = could not find container \"46e44e5d19240a478c4a2a4b0e82b2d3b0f7f093805adfc431f8bf89705aa7d7\": container with ID starting with 46e44e5d19240a478c4a2a4b0e82b2d3b0f7f093805adfc431f8bf89705aa7d7 not found: ID does not exist" Nov 03 21:48:38.964244 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.964201 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr"] Nov 03 21:48:38.970213 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:38.970166 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-pzgqr"] Nov 03 21:48:39.007641 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:39.007604 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6"] Nov 03 21:48:39.020582 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:39.020498 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-r4xg6"] Nov 03 21:48:39.282245 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:39.282199 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:48:39.282776 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:39.282291 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" podUID="8ad94b74-be79-48fd-847d-491eb6d5943a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:48:39.948735 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:39.948364 5041 generic.go:356] "Generic (PLEG): container finished" podID="f738b002-3e77-4a00-9982-2cb525d164e6" containerID="85b11d0a2404d3902b7fda4df0d136d596953392540e87ec21dfd80040c158e5" exitCode=0 Nov 03 21:48:39.948735 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:39.948526 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" event={"ID":"f738b002-3e77-4a00-9982-2cb525d164e6","Type":"ContainerDied","Data":"85b11d0a2404d3902b7fda4df0d136d596953392540e87ec21dfd80040c158e5"} Nov 03 21:48:39.956124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:39.953908 5041 generic.go:356] "Generic (PLEG): container finished" podID="523ee582-4304-412b-8269-cf9cf716c8a2" containerID="3b579db11de64b67ef6d213911848487c7d69d88904546ee67b0dce15fbc75a4" exitCode=0 Nov 03 21:48:39.956124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:39.953998 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" event={"ID":"523ee582-4304-412b-8269-cf9cf716c8a2","Type":"ContainerDied","Data":"3b579db11de64b67ef6d213911848487c7d69d88904546ee67b0dce15fbc75a4"} Nov 03 21:48:39.956124 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:39.954056 5041 scope.go:122] "RemoveContainer" containerID="c28a0fb769cde3626a45a18072804d306f6007c6ed852a07ba252a3e66a99299" Nov 03 21:48:39.961844 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:39.961723 5041 generic.go:356] "Generic (PLEG): container finished" podID="8ad94b74-be79-48fd-847d-491eb6d5943a" containerID="90984b1f19703f54a6c67c35ab35f4cf8060884219525e7400acc6779b4d02d3" exitCode=0 Nov 03 21:48:39.962027 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:39.961942 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" event={"ID":"8ad94b74-be79-48fd-847d-491eb6d5943a","Type":"ContainerDied","Data":"90984b1f19703f54a6c67c35ab35f4cf8060884219525e7400acc6779b4d02d3"} Nov 03 21:48:39.965089 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:39.965056 5041 generic.go:356] "Generic (PLEG): container finished" podID="149765b9-f8c8-4896-9b15-79f567e82336" containerID="711f15b5b8f73ec510f5dda14ead4b1df6b852205a3442ae8b0609cc761678ee" exitCode=0 Nov 03 21:48:39.965407 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:39.965373 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" event={"ID":"149765b9-f8c8-4896-9b15-79f567e82336","Type":"ContainerDied","Data":"711f15b5b8f73ec510f5dda14ead4b1df6b852205a3442ae8b0609cc761678ee"} Nov 03 21:48:40.121957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.121927 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:48:40.246621 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.246569 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:48:40.265773 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.265746 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-istio-envoy\") pod \"f738b002-3e77-4a00-9982-2cb525d164e6\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " Nov 03 21:48:40.265949 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.265790 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/149765b9-f8c8-4896-9b15-79f567e82336-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/149765b9-f8c8-4896-9b15-79f567e82336-istio-podinfo\") pod \"149765b9-f8c8-4896-9b15-79f567e82336\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " Nov 03 21:48:40.265949 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.265821 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/149765b9-f8c8-4896-9b15-79f567e82336-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/149765b9-f8c8-4896-9b15-79f567e82336-istiod-ca-cert\") pod \"149765b9-f8c8-4896-9b15-79f567e82336\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " Nov 03 21:48:40.265949 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.265846 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-workload-socket\") pod \"f738b002-3e77-4a00-9982-2cb525d164e6\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " Nov 03 21:48:40.265949 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.265889 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/149765b9-f8c8-4896-9b15-79f567e82336-kube-api-access-bhxsv\" (UniqueName: \"kubernetes.io/projected/149765b9-f8c8-4896-9b15-79f567e82336-kube-api-access-bhxsv\") pod \"149765b9-f8c8-4896-9b15-79f567e82336\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " Nov 03 21:48:40.266173 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.265952 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-credential-socket\") pod \"149765b9-f8c8-4896-9b15-79f567e82336\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " Nov 03 21:48:40.266173 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266014 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-workload-certs\") pod \"149765b9-f8c8-4896-9b15-79f567e82336\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " Nov 03 21:48:40.266173 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266059 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-istio-data\") pod \"149765b9-f8c8-4896-9b15-79f567e82336\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " Nov 03 21:48:40.266173 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266097 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-istio-data\") pod \"f738b002-3e77-4a00-9982-2cb525d164e6\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " Nov 03 21:48:40.266173 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266168 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-workload-socket\") pod \"149765b9-f8c8-4896-9b15-79f567e82336\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " Nov 03 21:48:40.266433 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266169 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-workload-socket" pod "f738b002-3e77-4a00-9982-2cb525d164e6" (UID: "f738b002-3e77-4a00-9982-2cb525d164e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.266433 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266199 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-workload-certs\") pod \"f738b002-3e77-4a00-9982-2cb525d164e6\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " Nov 03 21:48:40.266433 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266243 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-credential-socket\") pod \"f738b002-3e77-4a00-9982-2cb525d164e6\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " Nov 03 21:48:40.266433 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266289 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f738b002-3e77-4a00-9982-2cb525d164e6-kube-api-access-9v2z5\" (UniqueName: \"kubernetes.io/projected/f738b002-3e77-4a00-9982-2cb525d164e6-kube-api-access-9v2z5\") pod \"f738b002-3e77-4a00-9982-2cb525d164e6\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " Nov 03 21:48:40.266433 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266336 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f738b002-3e77-4a00-9982-2cb525d164e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f738b002-3e77-4a00-9982-2cb525d164e6-istiod-ca-cert\") pod \"f738b002-3e77-4a00-9982-2cb525d164e6\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " Nov 03 21:48:40.266433 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266370 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f738b002-3e77-4a00-9982-2cb525d164e6-istio-token\" (UniqueName: \"kubernetes.io/projected/f738b002-3e77-4a00-9982-2cb525d164e6-istio-token\") pod \"f738b002-3e77-4a00-9982-2cb525d164e6\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " Nov 03 21:48:40.266433 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266397 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/149765b9-f8c8-4896-9b15-79f567e82336-istio-token\" (UniqueName: \"kubernetes.io/projected/149765b9-f8c8-4896-9b15-79f567e82336-istio-token\") pod \"149765b9-f8c8-4896-9b15-79f567e82336\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " Nov 03 21:48:40.266824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266438 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f738b002-3e77-4a00-9982-2cb525d164e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f738b002-3e77-4a00-9982-2cb525d164e6-istio-podinfo\") pod \"f738b002-3e77-4a00-9982-2cb525d164e6\" (UID: \"f738b002-3e77-4a00-9982-2cb525d164e6\") " Nov 03 21:48:40.266824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266497 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-istio-envoy\") pod \"149765b9-f8c8-4896-9b15-79f567e82336\" (UID: \"149765b9-f8c8-4896-9b15-79f567e82336\") " Nov 03 21:48:40.267021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.267002 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.268021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266333 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-credential-socket" pod "149765b9-f8c8-4896-9b15-79f567e82336" (UID: "149765b9-f8c8-4896-9b15-79f567e82336"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.268021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266357 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-workload-certs" pod "149765b9-f8c8-4896-9b15-79f567e82336" (UID: "149765b9-f8c8-4896-9b15-79f567e82336"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.268021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266363 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/149765b9-f8c8-4896-9b15-79f567e82336-istiod-ca-cert" pod "149765b9-f8c8-4896-9b15-79f567e82336" (UID: "149765b9-f8c8-4896-9b15-79f567e82336"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:40.268021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266477 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-istio-data" pod "f738b002-3e77-4a00-9982-2cb525d164e6" (UID: "f738b002-3e77-4a00-9982-2cb525d164e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.268021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266577 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-credential-socket" pod "f738b002-3e77-4a00-9982-2cb525d164e6" (UID: "f738b002-3e77-4a00-9982-2cb525d164e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.268021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266591 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-istio-data" pod "149765b9-f8c8-4896-9b15-79f567e82336" (UID: "149765b9-f8c8-4896-9b15-79f567e82336"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.268021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266829 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-workload-socket" pod "149765b9-f8c8-4896-9b15-79f567e82336" (UID: "149765b9-f8c8-4896-9b15-79f567e82336"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.268021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.266843 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f738b002-3e77-4a00-9982-2cb525d164e6-istiod-ca-cert" pod "f738b002-3e77-4a00-9982-2cb525d164e6" (UID: "f738b002-3e77-4a00-9982-2cb525d164e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:40.268021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.267000 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-workload-certs" pod "f738b002-3e77-4a00-9982-2cb525d164e6" (UID: "f738b002-3e77-4a00-9982-2cb525d164e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.268870 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.268823 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/149765b9-f8c8-4896-9b15-79f567e82336-kube-api-access-bhxsv" pod "149765b9-f8c8-4896-9b15-79f567e82336" (UID: "149765b9-f8c8-4896-9b15-79f567e82336"). InnerVolumeSpecName "kube-api-access-bhxsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:40.269632 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.269608 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f738b002-3e77-4a00-9982-2cb525d164e6-istio-podinfo" pod "f738b002-3e77-4a00-9982-2cb525d164e6" (UID: "f738b002-3e77-4a00-9982-2cb525d164e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:40.269862 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.269823 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f738b002-3e77-4a00-9982-2cb525d164e6-kube-api-access-9v2z5" pod "f738b002-3e77-4a00-9982-2cb525d164e6" (UID: "f738b002-3e77-4a00-9982-2cb525d164e6"). InnerVolumeSpecName "kube-api-access-9v2z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:40.270119 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.270079 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/149765b9-f8c8-4896-9b15-79f567e82336-istio-podinfo" pod "149765b9-f8c8-4896-9b15-79f567e82336" (UID: "149765b9-f8c8-4896-9b15-79f567e82336"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:40.270119 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.270097 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f738b002-3e77-4a00-9982-2cb525d164e6-istio-token" pod "f738b002-3e77-4a00-9982-2cb525d164e6" (UID: "f738b002-3e77-4a00-9982-2cb525d164e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:40.270460 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.270420 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-istio-envoy" pod "149765b9-f8c8-4896-9b15-79f567e82336" (UID: "149765b9-f8c8-4896-9b15-79f567e82336"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.271166 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.271144 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-istio-envoy" pod "f738b002-3e77-4a00-9982-2cb525d164e6" (UID: "f738b002-3e77-4a00-9982-2cb525d164e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.271237 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.271224 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/149765b9-f8c8-4896-9b15-79f567e82336-istio-token" pod "149765b9-f8c8-4896-9b15-79f567e82336" (UID: "149765b9-f8c8-4896-9b15-79f567e82336"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:40.274048 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.274026 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:48:40.300888 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.294638 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:48:40.368352 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.368299 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-envoy\") pod \"8ad94b74-be79-48fd-847d-491eb6d5943a\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " Nov 03 21:48:40.368352 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.368358 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-istio-envoy\") pod \"523ee582-4304-412b-8269-cf9cf716c8a2\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " Nov 03 21:48:40.368629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.368403 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-credential-socket\") pod \"523ee582-4304-412b-8269-cf9cf716c8a2\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " Nov 03 21:48:40.368629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.368465 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-podinfo\") pod \"8ad94b74-be79-48fd-847d-491eb6d5943a\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " Nov 03 21:48:40.368629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.368498 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-workload-socket\") pod \"8ad94b74-be79-48fd-847d-491eb6d5943a\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " Nov 03 21:48:40.368629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.368570 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523ee582-4304-412b-8269-cf9cf716c8a2-istio-token\" (UniqueName: \"kubernetes.io/projected/523ee582-4304-412b-8269-cf9cf716c8a2-istio-token\") pod \"523ee582-4304-412b-8269-cf9cf716c8a2\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " Nov 03 21:48:40.368629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.368600 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-workload-certs\") pod \"523ee582-4304-412b-8269-cf9cf716c8a2\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " Nov 03 21:48:40.368944 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.368631 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad94b74-be79-48fd-847d-491eb6d5943a-kube-api-access-6qrss\" (UniqueName: \"kubernetes.io/projected/8ad94b74-be79-48fd-847d-491eb6d5943a-kube-api-access-6qrss\") pod \"8ad94b74-be79-48fd-847d-491eb6d5943a\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " Nov 03 21:48:40.368944 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.368710 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-credential-socket" pod "523ee582-4304-412b-8269-cf9cf716c8a2" (UID: "523ee582-4304-412b-8269-cf9cf716c8a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.368944 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.368726 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-credential-socket\") pod \"8ad94b74-be79-48fd-847d-491eb6d5943a\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " Nov 03 21:48:40.368944 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.368802 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/523ee582-4304-412b-8269-cf9cf716c8a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523ee582-4304-412b-8269-cf9cf716c8a2-istiod-ca-cert\") pod \"523ee582-4304-412b-8269-cf9cf716c8a2\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " Nov 03 21:48:40.368944 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.368842 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523ee582-4304-412b-8269-cf9cf716c8a2-kube-api-access-qfps4\" (UniqueName: \"kubernetes.io/projected/523ee582-4304-412b-8269-cf9cf716c8a2-kube-api-access-qfps4\") pod \"523ee582-4304-412b-8269-cf9cf716c8a2\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " Nov 03 21:48:40.368944 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.368893 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-workload-certs\") pod \"8ad94b74-be79-48fd-847d-491eb6d5943a\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " Nov 03 21:48:40.369257 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.368982 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-credential-socket" pod "8ad94b74-be79-48fd-847d-491eb6d5943a" (UID: "8ad94b74-be79-48fd-847d-491eb6d5943a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.369257 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.368995 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-workload-socket" pod "8ad94b74-be79-48fd-847d-491eb6d5943a" (UID: "8ad94b74-be79-48fd-847d-491eb6d5943a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.369257 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.369006 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-workload-socket\") pod \"523ee582-4304-412b-8269-cf9cf716c8a2\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " Nov 03 21:48:40.369257 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.369212 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-workload-socket" pod "523ee582-4304-412b-8269-cf9cf716c8a2" (UID: "523ee582-4304-412b-8269-cf9cf716c8a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.369463 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/523ee582-4304-412b-8269-cf9cf716c8a2-istiod-ca-cert" pod "523ee582-4304-412b-8269-cf9cf716c8a2" (UID: "523ee582-4304-412b-8269-cf9cf716c8a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.370085 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-workload-certs" pod "8ad94b74-be79-48fd-847d-491eb6d5943a" (UID: "8ad94b74-be79-48fd-847d-491eb6d5943a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.370135 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-token\" (UniqueName: \"kubernetes.io/projected/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-token\") pod \"8ad94b74-be79-48fd-847d-491eb6d5943a\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.370232 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-data\") pod \"8ad94b74-be79-48fd-847d-491eb6d5943a\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.370292 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad94b74-be79-48fd-847d-491eb6d5943a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad94b74-be79-48fd-847d-491eb6d5943a-istiod-ca-cert\") pod \"8ad94b74-be79-48fd-847d-491eb6d5943a\" (UID: \"8ad94b74-be79-48fd-847d-491eb6d5943a\") " Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.370324 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-istio-data\") pod \"523ee582-4304-412b-8269-cf9cf716c8a2\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.370356 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/523ee582-4304-412b-8269-cf9cf716c8a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523ee582-4304-412b-8269-cf9cf716c8a2-istio-podinfo\") pod \"523ee582-4304-412b-8269-cf9cf716c8a2\" (UID: \"523ee582-4304-412b-8269-cf9cf716c8a2\") " Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.370974 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.370998 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371014 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371029 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371043 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523ee582-4304-412b-8269-cf9cf716c8a2-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371057 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v2z5\" (UniqueName: \"kubernetes.io/projected/f738b002-3e77-4a00-9982-2cb525d164e6-kube-api-access-9v2z5\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371069 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f738b002-3e77-4a00-9982-2cb525d164e6-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371081 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371093 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f738b002-3e77-4a00-9982-2cb525d164e6-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371106 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/149765b9-f8c8-4896-9b15-79f567e82336-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371114 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f738b002-3e77-4a00-9982-2cb525d164e6-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371122 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371134 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371150 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371162 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/149765b9-f8c8-4896-9b15-79f567e82336-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371174 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/149765b9-f8c8-4896-9b15-79f567e82336-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371188 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhxsv\" (UniqueName: \"kubernetes.io/projected/149765b9-f8c8-4896-9b15-79f567e82336-kube-api-access-bhxsv\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371201 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371214 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371228 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371241 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371253 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/149765b9-f8c8-4896-9b15-79f567e82336-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371265 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f738b002-3e77-4a00-9982-2cb525d164e6-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371507 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-envoy" pod "8ad94b74-be79-48fd-847d-491eb6d5943a" (UID: "8ad94b74-be79-48fd-847d-491eb6d5943a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371542 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-podinfo" pod "8ad94b74-be79-48fd-847d-491eb6d5943a" (UID: "8ad94b74-be79-48fd-847d-491eb6d5943a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371558 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-workload-certs" pod "523ee582-4304-412b-8269-cf9cf716c8a2" (UID: "523ee582-4304-412b-8269-cf9cf716c8a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371574 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-istio-envoy" pod "523ee582-4304-412b-8269-cf9cf716c8a2" (UID: "523ee582-4304-412b-8269-cf9cf716c8a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371806 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-data" pod "8ad94b74-be79-48fd-847d-491eb6d5943a" (UID: "8ad94b74-be79-48fd-847d-491eb6d5943a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.371891 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-istio-data" pod "523ee582-4304-412b-8269-cf9cf716c8a2" (UID: "523ee582-4304-412b-8269-cf9cf716c8a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:40.372046 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.372000 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad94b74-be79-48fd-847d-491eb6d5943a-istiod-ca-cert" pod "8ad94b74-be79-48fd-847d-491eb6d5943a" (UID: "8ad94b74-be79-48fd-847d-491eb6d5943a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:40.373935 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.372606 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523ee582-4304-412b-8269-cf9cf716c8a2-istio-token" pod "523ee582-4304-412b-8269-cf9cf716c8a2" (UID: "523ee582-4304-412b-8269-cf9cf716c8a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:40.373935 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.372672 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad94b74-be79-48fd-847d-491eb6d5943a-kube-api-access-6qrss" pod "8ad94b74-be79-48fd-847d-491eb6d5943a" (UID: "8ad94b74-be79-48fd-847d-491eb6d5943a"). InnerVolumeSpecName "kube-api-access-6qrss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:40.373935 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.373528 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/523ee582-4304-412b-8269-cf9cf716c8a2-istio-podinfo" pod "523ee582-4304-412b-8269-cf9cf716c8a2" (UID: "523ee582-4304-412b-8269-cf9cf716c8a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:40.373935 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.373628 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523ee582-4304-412b-8269-cf9cf716c8a2-kube-api-access-qfps4" pod "523ee582-4304-412b-8269-cf9cf716c8a2" (UID: "523ee582-4304-412b-8269-cf9cf716c8a2"). InnerVolumeSpecName "kube-api-access-qfps4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:40.374080 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.374062 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-token" pod "8ad94b74-be79-48fd-847d-491eb6d5943a" (UID: "8ad94b74-be79-48fd-847d-491eb6d5943a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:40.472242 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.472198 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.472242 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.472236 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.472242 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.472246 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad94b74-be79-48fd-847d-491eb6d5943a-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.472499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.472255 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.472499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.472264 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523ee582-4304-412b-8269-cf9cf716c8a2-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.472499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.472274 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.472499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.472283 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.472499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.472290 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad94b74-be79-48fd-847d-491eb6d5943a-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.472499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.472298 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523ee582-4304-412b-8269-cf9cf716c8a2-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.472499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.472308 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523ee582-4304-412b-8269-cf9cf716c8a2-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.472499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.472317 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qrss\" (UniqueName: \"kubernetes.io/projected/8ad94b74-be79-48fd-847d-491eb6d5943a-kube-api-access-6qrss\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.472499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.472327 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfps4\" (UniqueName: \"kubernetes.io/projected/523ee582-4304-412b-8269-cf9cf716c8a2-kube-api-access-qfps4\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:40.623307 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.623217 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d409dea-7691-4be3-9769-5e196850a3f2" path="/var/lib/kubelet/pods/7d409dea-7691-4be3-9769-5e196850a3f2/volumes" Nov 03 21:48:40.623788 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.623769 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" path="/var/lib/kubelet/pods/fcda3325-5e04-4611-9a1e-48075b67d03b/volumes" Nov 03 21:48:40.780617 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:48:40.780574 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:48:40.977569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.977535 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" Nov 03 21:48:40.977569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.977557 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd" event={"ID":"149765b9-f8c8-4896-9b15-79f567e82336","Type":"ContainerDied","Data":"2e1719dafd31ea9b78ecb4e3cbac83ec13590eb7c45581222fd933d592934153"} Nov 03 21:48:40.978161 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.977617 5041 scope.go:122] "RemoveContainer" containerID="711f15b5b8f73ec510f5dda14ead4b1df6b852205a3442ae8b0609cc761678ee" Nov 03 21:48:40.979523 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.979503 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" Nov 03 21:48:40.979807 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.979550 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg" event={"ID":"f738b002-3e77-4a00-9982-2cb525d164e6","Type":"ContainerDied","Data":"8ccf964d0453c65561a05cfd52897ed82b765147372293bc96bbe62a6cd4ff0a"} Nov 03 21:48:40.981517 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.981498 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" Nov 03 21:48:40.981710 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.981512 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" event={"ID":"523ee582-4304-412b-8269-cf9cf716c8a2","Type":"ContainerDied","Data":"428e6e3e9c745cce28f5861cc6070c37c0e819c4b1a4f410eb17a2a3a7cd5f28"} Nov 03 21:48:40.983313 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.983288 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" event={"ID":"8ad94b74-be79-48fd-847d-491eb6d5943a","Type":"ContainerDied","Data":"4d5b44da487dfdc4ce23175f280ccc1ed1b2849912a30b0e8e451b39b4a5462d"} Nov 03 21:48:40.983403 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.983299 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4" Nov 03 21:48:40.992108 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:40.992086 5041 scope.go:122] "RemoveContainer" containerID="85b11d0a2404d3902b7fda4df0d136d596953392540e87ec21dfd80040c158e5" Nov 03 21:48:41.003676 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:41.003637 5041 scope.go:122] "RemoveContainer" containerID="3b579db11de64b67ef6d213911848487c7d69d88904546ee67b0dce15fbc75a4" Nov 03 21:48:41.015769 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:41.015692 5041 scope.go:122] "RemoveContainer" containerID="90984b1f19703f54a6c67c35ab35f4cf8060884219525e7400acc6779b4d02d3" Nov 03 21:48:41.049356 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:41.049300 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd"] Nov 03 21:48:41.056123 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:41.056090 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e246f9-openshift-default-65c46599d9-s8czd"] Nov 03 21:48:41.081507 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:41.081467 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4"] Nov 03 21:48:41.086840 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:41.086804 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-8d4q4"] Nov 03 21:48:41.120026 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:41.119995 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg"] Nov 03 21:48:41.125394 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:41.125355 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-fswxg"] Nov 03 21:48:41.160791 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:41.160746 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k"] Nov 03 21:48:41.166054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:41.166021 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k"] Nov 03 21:48:41.259901 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:41.259805 5041 patch_prober.go:29] interesting pod/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:48:41.259901 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:41.259867 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-05bc27-openshift-default-6df586d6ff-9rm5k" podUID="523ee582-4304-412b-8269-cf9cf716c8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:48:41.266982 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:41.266958 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:42.628689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:42.628567 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="149765b9-f8c8-4896-9b15-79f567e82336" path="/var/lib/kubelet/pods/149765b9-f8c8-4896-9b15-79f567e82336/volumes" Nov 03 21:48:42.629486 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:42.629462 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="523ee582-4304-412b-8269-cf9cf716c8a2" path="/var/lib/kubelet/pods/523ee582-4304-412b-8269-cf9cf716c8a2/volumes" Nov 03 21:48:42.630215 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:42.630190 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad94b74-be79-48fd-847d-491eb6d5943a" path="/var/lib/kubelet/pods/8ad94b74-be79-48fd-847d-491eb6d5943a/volumes" Nov 03 21:48:42.631334 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:42.631312 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f738b002-3e77-4a00-9982-2cb525d164e6" path="/var/lib/kubelet/pods/f738b002-3e77-4a00-9982-2cb525d164e6/volumes" Nov 03 21:48:43.388489 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.388452 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj"] Nov 03 21:48:43.389080 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389066 5041 cpu_manager.go:414] "containerMap: removing container" podUID="d87e1039-2359-4a69-a9ad-c029d9987a01" containerName="istio-proxy" Nov 03 21:48:43.389131 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389081 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87e1039-2359-4a69-a9ad-c029d9987a01" containerName="istio-proxy" Nov 03 21:48:43.389131 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389094 5041 cpu_manager.go:414] "containerMap: removing container" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" Nov 03 21:48:43.389131 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389100 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" Nov 03 21:48:43.389131 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389116 5041 cpu_manager.go:414] "containerMap: removing container" podUID="7d409dea-7691-4be3-9769-5e196850a3f2" containerName="istio-proxy" Nov 03 21:48:43.389131 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389122 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d409dea-7691-4be3-9769-5e196850a3f2" containerName="istio-proxy" Nov 03 21:48:43.389131 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389130 5041 cpu_manager.go:414] "containerMap: removing container" podUID="8ad94b74-be79-48fd-847d-491eb6d5943a" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389136 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad94b74-be79-48fd-847d-491eb6d5943a" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389156 5041 cpu_manager.go:414] "containerMap: removing container" podUID="e29a22a0-dc41-4a86-ac9f-d6392e6c5472" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389161 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29a22a0-dc41-4a86-ac9f-d6392e6c5472" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389177 5041 cpu_manager.go:414] "containerMap: removing container" podUID="e9cb2586-4e9a-47cf-abd8-604c0f1264d5" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389184 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cb2586-4e9a-47cf-abd8-604c0f1264d5" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389196 5041 cpu_manager.go:414] "containerMap: removing container" podUID="523ee582-4304-412b-8269-cf9cf716c8a2" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389201 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523ee582-4304-412b-8269-cf9cf716c8a2" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389210 5041 cpu_manager.go:414] "containerMap: removing container" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389214 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389222 5041 cpu_manager.go:414] "containerMap: removing container" podUID="f738b002-3e77-4a00-9982-2cb525d164e6" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389226 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f738b002-3e77-4a00-9982-2cb525d164e6" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389232 5041 cpu_manager.go:414] "containerMap: removing container" podUID="72a848ed-b60e-4690-8933-8abdcac1d8ba" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389236 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a848ed-b60e-4690-8933-8abdcac1d8ba" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389245 5041 cpu_manager.go:414] "containerMap: removing container" podUID="557fc6e5-610e-457d-895c-fa9746131887" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389249 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="557fc6e5-610e-457d-895c-fa9746131887" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389255 5041 cpu_manager.go:414] "containerMap: removing container" podUID="523ee582-4304-412b-8269-cf9cf716c8a2" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389260 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523ee582-4304-412b-8269-cf9cf716c8a2" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389267 5041 cpu_manager.go:414] "containerMap: removing container" podUID="ca8a2829-0a62-49af-b6ba-9546aa670350" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389271 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca8a2829-0a62-49af-b6ba-9546aa670350" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389277 5041 cpu_manager.go:414] "containerMap: removing container" podUID="bc499793-421d-4550-af6d-691af2b07aa9" containerName="istio-proxy" Nov 03 21:48:43.389316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389282 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc499793-421d-4550-af6d-691af2b07aa9" containerName="istio-proxy" Nov 03 21:48:43.389948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389360 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a848ed-b60e-4690-8933-8abdcac1d8ba" containerName="istio-proxy" Nov 03 21:48:43.389948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389370 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="557fc6e5-610e-457d-895c-fa9746131887" containerName="istio-proxy" Nov 03 21:48:43.389948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389376 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="523ee582-4304-412b-8269-cf9cf716c8a2" containerName="istio-proxy" Nov 03 21:48:43.389948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389382 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca8a2829-0a62-49af-b6ba-9546aa670350" containerName="istio-proxy" Nov 03 21:48:43.389948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389386 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc499793-421d-4550-af6d-691af2b07aa9" containerName="istio-proxy" Nov 03 21:48:43.389948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389395 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87e1039-2359-4a69-a9ad-c029d9987a01" containerName="istio-proxy" Nov 03 21:48:43.389948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389401 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcda3325-5e04-4611-9a1e-48075b67d03b" containerName="istio-proxy" Nov 03 21:48:43.389948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389409 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d409dea-7691-4be3-9769-5e196850a3f2" containerName="istio-proxy" Nov 03 21:48:43.389948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389415 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad94b74-be79-48fd-847d-491eb6d5943a" containerName="istio-proxy" Nov 03 21:48:43.389948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389422 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29a22a0-dc41-4a86-ac9f-d6392e6c5472" containerName="istio-proxy" Nov 03 21:48:43.389948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389429 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cb2586-4e9a-47cf-abd8-604c0f1264d5" containerName="istio-proxy" Nov 03 21:48:43.389948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389435 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="523ee582-4304-412b-8269-cf9cf716c8a2" containerName="istio-proxy" Nov 03 21:48:43.389948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389441 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="149765b9-f8c8-4896-9b15-79f567e82336" containerName="istio-proxy" Nov 03 21:48:43.389948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.389446 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f738b002-3e77-4a00-9982-2cb525d164e6" containerName="istio-proxy" Nov 03 21:48:43.395245 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.395199 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj" Nov 03 21:48:43.409781 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.409738 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28004d42-99bd-44a3-9e2e-94c4189965f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-517149-zpddj\" (UID: \"28004d42-99bd-44a3-9e2e-94c4189965f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj" Nov 03 21:48:43.409955 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.409851 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcs7f\" (UniqueName: \"kubernetes.io/projected/28004d42-99bd-44a3-9e2e-94c4189965f0-kube-api-access-bcs7f\") pod \"maas-api-revoke-keys-e2e-worker-w6-517149-zpddj\" (UID: \"28004d42-99bd-44a3-9e2e-94c4189965f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj" Nov 03 21:48:43.409955 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.409909 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28004d42-99bd-44a3-9e2e-94c4189965f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-517149-zpddj\" (UID: \"28004d42-99bd-44a3-9e2e-94c4189965f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj" Nov 03 21:48:43.415810 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.415779 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj"] Nov 03 21:48:43.427730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.419276 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:48:43.427730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.419323 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:48:43.511838 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.511795 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28004d42-99bd-44a3-9e2e-94c4189965f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-517149-zpddj\" (UID: \"28004d42-99bd-44a3-9e2e-94c4189965f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj" Nov 03 21:48:43.512044 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.511921 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcs7f\" (UniqueName: \"kubernetes.io/projected/28004d42-99bd-44a3-9e2e-94c4189965f0-kube-api-access-bcs7f\") pod \"maas-api-revoke-keys-e2e-worker-w6-517149-zpddj\" (UID: \"28004d42-99bd-44a3-9e2e-94c4189965f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj" Nov 03 21:48:43.512044 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.511987 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28004d42-99bd-44a3-9e2e-94c4189965f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-517149-zpddj\" (UID: \"28004d42-99bd-44a3-9e2e-94c4189965f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj" Nov 03 21:48:43.512866 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.512838 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28004d42-99bd-44a3-9e2e-94c4189965f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-517149-zpddj\" (UID: \"28004d42-99bd-44a3-9e2e-94c4189965f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj" Nov 03 21:48:43.515770 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.515739 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28004d42-99bd-44a3-9e2e-94c4189965f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-517149-zpddj\" (UID: \"28004d42-99bd-44a3-9e2e-94c4189965f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj" Nov 03 21:48:43.536257 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.536211 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcs7f\" (UniqueName: \"kubernetes.io/projected/28004d42-99bd-44a3-9e2e-94c4189965f0-kube-api-access-bcs7f\") pod \"maas-api-revoke-keys-e2e-worker-w6-517149-zpddj\" (UID: \"28004d42-99bd-44a3-9e2e-94c4189965f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj" Nov 03 21:48:43.713500 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.713466 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj" Nov 03 21:48:43.753403 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.753369 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.820915 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-envoy\") pod \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.820996 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istiod-ca-cert\") pod \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.821045 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-token\" (UniqueName: \"kubernetes.io/projected/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-token\") pod \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.821095 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-data\") pod \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.821145 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-kube-api-access-bfchs\" (UniqueName: \"kubernetes.io/projected/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-kube-api-access-bfchs\") pod \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.821256 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-workload-certs\") pod \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.821322 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-credential-socket\") pod \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.821419 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-workload-socket\") pod \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.821590 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-podinfo\") pod \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\" (UID: \"668e0b57-1f4e-441f-8d8b-fddc8524fcf6\") " Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.822355 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-data" pod "668e0b57-1f4e-441f-8d8b-fddc8524fcf6" (UID: "668e0b57-1f4e-441f-8d8b-fddc8524fcf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.822636 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.825246 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-kube-api-access-bfchs" pod "668e0b57-1f4e-441f-8d8b-fddc8524fcf6" (UID: "668e0b57-1f4e-441f-8d8b-fddc8524fcf6"). InnerVolumeSpecName "kube-api-access-bfchs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.825466 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-workload-certs" pod "668e0b57-1f4e-441f-8d8b-fddc8524fcf6" (UID: "668e0b57-1f4e-441f-8d8b-fddc8524fcf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.825633 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-credential-socket" pod "668e0b57-1f4e-441f-8d8b-fddc8524fcf6" (UID: "668e0b57-1f4e-441f-8d8b-fddc8524fcf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.825912 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-workload-socket" pod "668e0b57-1f4e-441f-8d8b-fddc8524fcf6" (UID: "668e0b57-1f4e-441f-8d8b-fddc8524fcf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.828483 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-podinfo" pod "668e0b57-1f4e-441f-8d8b-fddc8524fcf6" (UID: "668e0b57-1f4e-441f-8d8b-fddc8524fcf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:43.830681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.828790 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istiod-ca-cert" pod "668e0b57-1f4e-441f-8d8b-fddc8524fcf6" (UID: "668e0b57-1f4e-441f-8d8b-fddc8524fcf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:43.835570 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.832352 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-token" pod "668e0b57-1f4e-441f-8d8b-fddc8524fcf6" (UID: "668e0b57-1f4e-441f-8d8b-fddc8524fcf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:43.835570 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.833159 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-envoy" pod "668e0b57-1f4e-441f-8d8b-fddc8524fcf6" (UID: "668e0b57-1f4e-441f-8d8b-fddc8524fcf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:43.924279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.924237 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:43.924279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.924272 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:43.924279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.924282 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:43.924617 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.924292 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfchs\" (UniqueName: \"kubernetes.io/projected/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-kube-api-access-bfchs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:43.924617 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.924303 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:43.924617 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.924311 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:43.924617 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.924321 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:43.924617 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:43.924329 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668e0b57-1f4e-441f-8d8b-fddc8524fcf6-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:44.025015 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:44.024979 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj"] Nov 03 21:48:44.027550 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:44.027518 5041 generic.go:356] "Generic (PLEG): container finished" podID="668e0b57-1f4e-441f-8d8b-fddc8524fcf6" containerID="6661c448bdef4714bdb8408c1ecdec3bdb28ee4dd65116a54092077b1f8d0de8" exitCode=0 Nov 03 21:48:44.027813 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:48:44.027782 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28004d42_99bd_44a3_9e2e_94c4189965f0.slice/crio-e8035011ec29e735af636602ab96fc533bc19990692a05fdc728c950aef8e993 WatchSource:0}: Error finding container e8035011ec29e735af636602ab96fc533bc19990692a05fdc728c950aef8e993: Status 404 returned error can't find the container with id e8035011ec29e735af636602ab96fc533bc19990692a05fdc728c950aef8e993 Nov 03 21:48:44.027813 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:44.027791 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" event={"ID":"668e0b57-1f4e-441f-8d8b-fddc8524fcf6","Type":"ContainerDied","Data":"6661c448bdef4714bdb8408c1ecdec3bdb28ee4dd65116a54092077b1f8d0de8"} Nov 03 21:48:44.028098 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:44.027824 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" Nov 03 21:48:44.028098 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:44.027860 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj" event={"ID":"668e0b57-1f4e-441f-8d8b-fddc8524fcf6","Type":"ContainerDied","Data":"79909632d901238ca6c9ab3f6ed7974bf751a561fffe66955217a0dee7e1a666"} Nov 03 21:48:44.028098 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:44.027897 5041 scope.go:122] "RemoveContainer" containerID="6661c448bdef4714bdb8408c1ecdec3bdb28ee4dd65116a54092077b1f8d0de8" Nov 03 21:48:44.052694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:44.052655 5041 scope.go:122] "RemoveContainer" containerID="6661c448bdef4714bdb8408c1ecdec3bdb28ee4dd65116a54092077b1f8d0de8" Nov 03 21:48:44.053046 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:48:44.053019 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6661c448bdef4714bdb8408c1ecdec3bdb28ee4dd65116a54092077b1f8d0de8\": container with ID starting with 6661c448bdef4714bdb8408c1ecdec3bdb28ee4dd65116a54092077b1f8d0de8 not found: ID does not exist" containerID="6661c448bdef4714bdb8408c1ecdec3bdb28ee4dd65116a54092077b1f8d0de8" Nov 03 21:48:44.053135 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:44.053053 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6661c448bdef4714bdb8408c1ecdec3bdb28ee4dd65116a54092077b1f8d0de8"} err="failed to get container status \"6661c448bdef4714bdb8408c1ecdec3bdb28ee4dd65116a54092077b1f8d0de8\": rpc error: code = NotFound desc = could not find container \"6661c448bdef4714bdb8408c1ecdec3bdb28ee4dd65116a54092077b1f8d0de8\": container with ID starting with 6661c448bdef4714bdb8408c1ecdec3bdb28ee4dd65116a54092077b1f8d0de8 not found: ID does not exist" Nov 03 21:48:44.117687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:44.111952 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj"] Nov 03 21:48:44.126743 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:44.122153 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a6da1-openshift-default-5d8fb4b669-pbrwj"] Nov 03 21:48:44.623083 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:44.623041 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="668e0b57-1f4e-441f-8d8b-fddc8524fcf6" path="/var/lib/kubelet/pods/668e0b57-1f4e-441f-8d8b-fddc8524fcf6/volumes" Nov 03 21:48:45.041391 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:45.041341 5041 generic.go:356] "Generic (PLEG): container finished" podID="28004d42-99bd-44a3-9e2e-94c4189965f0" containerID="aad69d224ec50e56097f4f228828f66374683b9a84103357860a1589219fc2f8" exitCode=0 Nov 03 21:48:45.041952 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:45.041419 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj" event={"ID":"28004d42-99bd-44a3-9e2e-94c4189965f0","Type":"ContainerDied","Data":"aad69d224ec50e56097f4f228828f66374683b9a84103357860a1589219fc2f8"} Nov 03 21:48:45.041952 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:45.041476 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj" event={"ID":"28004d42-99bd-44a3-9e2e-94c4189965f0","Type":"ContainerStarted","Data":"e8035011ec29e735af636602ab96fc533bc19990692a05fdc728c950aef8e993"} Nov 03 21:48:45.860248 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:45.857909 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr"] Nov 03 21:48:45.860248 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:45.858171 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" podUID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" containerName="istio-proxy" containerID="cri-o://6491ab9a2d3123365fcf19697178a8895552eae48eb6fb8dadef33e6f11ffff2" gracePeriod=30 Nov 03 21:48:45.869617 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:45.869586 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6"] Nov 03 21:48:45.869887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:45.869850 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" podUID="6ed859d9-82ee-46b1-9e96-776b7917656d" containerName="istio-proxy" containerID="cri-o://26b7768609b866b137d560a15c95e5b6978bb2181ac1affd75a2f3183cae6afa" gracePeriod=30 Nov 03 21:48:45.875523 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:45.875491 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs"] Nov 03 21:48:45.875764 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:45.875739 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" containerID="cri-o://ce79be379e644fad8ba3c9f4ce0b02c78ab6876da16be67adedb10dad3a219db" gracePeriod=30 Nov 03 21:48:45.881821 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:45.881799 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn"] Nov 03 21:48:45.882029 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:45.882010 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" containerID="cri-o://d817f1837d733f59ff5a94fc2062e5a0b8e66c325c3c44d983b2734062e5c71a" gracePeriod=30 Nov 03 21:48:46.216228 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:46.216202 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj" Nov 03 21:48:46.266830 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:46.266795 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28004d42-99bd-44a3-9e2e-94c4189965f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28004d42-99bd-44a3-9e2e-94c4189965f0-maas-api-openshift-ca\") pod \"28004d42-99bd-44a3-9e2e-94c4189965f0\" (UID: \"28004d42-99bd-44a3-9e2e-94c4189965f0\") " Nov 03 21:48:46.267012 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:46.266927 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28004d42-99bd-44a3-9e2e-94c4189965f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28004d42-99bd-44a3-9e2e-94c4189965f0-maas-api-xks-ca\") pod \"28004d42-99bd-44a3-9e2e-94c4189965f0\" (UID: \"28004d42-99bd-44a3-9e2e-94c4189965f0\") " Nov 03 21:48:46.267097 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:46.267078 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28004d42-99bd-44a3-9e2e-94c4189965f0-kube-api-access-bcs7f\" (UniqueName: \"kubernetes.io/projected/28004d42-99bd-44a3-9e2e-94c4189965f0-kube-api-access-bcs7f\") pod \"28004d42-99bd-44a3-9e2e-94c4189965f0\" (UID: \"28004d42-99bd-44a3-9e2e-94c4189965f0\") " Nov 03 21:48:46.267203 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:46.267159 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28004d42-99bd-44a3-9e2e-94c4189965f0-maas-api-openshift-ca" pod "28004d42-99bd-44a3-9e2e-94c4189965f0" (UID: "28004d42-99bd-44a3-9e2e-94c4189965f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:46.267579 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:46.267557 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28004d42-99bd-44a3-9e2e-94c4189965f0-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:46.269478 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:46.269448 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28004d42-99bd-44a3-9e2e-94c4189965f0-maas-api-xks-ca" pod "28004d42-99bd-44a3-9e2e-94c4189965f0" (UID: "28004d42-99bd-44a3-9e2e-94c4189965f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:46.269576 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:46.269479 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28004d42-99bd-44a3-9e2e-94c4189965f0-kube-api-access-bcs7f" pod "28004d42-99bd-44a3-9e2e-94c4189965f0" (UID: "28004d42-99bd-44a3-9e2e-94c4189965f0"). InnerVolumeSpecName "kube-api-access-bcs7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:46.369450 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:46.369364 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcs7f\" (UniqueName: \"kubernetes.io/projected/28004d42-99bd-44a3-9e2e-94c4189965f0-kube-api-access-bcs7f\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:46.369450 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:46.369400 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28004d42-99bd-44a3-9e2e-94c4189965f0-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:47.062562 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:47.062529 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj" Nov 03 21:48:47.062562 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:47.062552 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj" event={"ID":"28004d42-99bd-44a3-9e2e-94c4189965f0","Type":"ContainerDied","Data":"e8035011ec29e735af636602ab96fc533bc19990692a05fdc728c950aef8e993"} Nov 03 21:48:47.062806 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:47.062605 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8035011ec29e735af636602ab96fc533bc19990692a05fdc728c950aef8e993" Nov 03 21:48:48.130478 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:48.130442 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz"] Nov 03 21:48:48.142197 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:48.142167 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5"] Nov 03 21:48:50.259112 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:50.259066 5041 patch_prober.go:29] interesting pod/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:48:50.259539 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:50.259164 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" podUID="6ed859d9-82ee-46b1-9e96-776b7917656d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:48:51.110887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.108349 5041 generic.go:356] "Generic (PLEG): container finished" podID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerID="ce79be379e644fad8ba3c9f4ce0b02c78ab6876da16be67adedb10dad3a219db" exitCode=0 Nov 03 21:48:51.110887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.108456 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" event={"ID":"367832c5-176f-4416-afd0-5a1bdb2b7570","Type":"ContainerDied","Data":"ce79be379e644fad8ba3c9f4ce0b02c78ab6876da16be67adedb10dad3a219db"} Nov 03 21:48:51.111114 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.111066 5041 generic.go:356] "Generic (PLEG): container finished" podID="6ed859d9-82ee-46b1-9e96-776b7917656d" containerID="26b7768609b866b137d560a15c95e5b6978bb2181ac1affd75a2f3183cae6afa" exitCode=0 Nov 03 21:48:51.111323 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.111265 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" event={"ID":"6ed859d9-82ee-46b1-9e96-776b7917656d","Type":"ContainerDied","Data":"26b7768609b866b137d560a15c95e5b6978bb2181ac1affd75a2f3183cae6afa"} Nov 03 21:48:51.114201 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.114180 5041 generic.go:356] "Generic (PLEG): container finished" podID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" containerID="6491ab9a2d3123365fcf19697178a8895552eae48eb6fb8dadef33e6f11ffff2" exitCode=0 Nov 03 21:48:51.114294 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.114270 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" event={"ID":"4b3eac22-c52b-48e7-bc7e-eddebf306cb5","Type":"ContainerDied","Data":"6491ab9a2d3123365fcf19697178a8895552eae48eb6fb8dadef33e6f11ffff2"} Nov 03 21:48:51.114356 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.114342 5041 scope.go:122] "RemoveContainer" containerID="8b7b7e6859be7f4cf7c3c39d92f79693b071ca823f9dc7ea42d91ebc41a30b1e" Nov 03 21:48:51.117375 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.117348 5041 generic.go:356] "Generic (PLEG): container finished" podID="49882dd3-115a-4449-b98d-9773358608bc" containerID="d817f1837d733f59ff5a94fc2062e5a0b8e66c325c3c44d983b2734062e5c71a" exitCode=0 Nov 03 21:48:51.117466 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.117436 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" event={"ID":"49882dd3-115a-4449-b98d-9773358608bc","Type":"ContainerDied","Data":"d817f1837d733f59ff5a94fc2062e5a0b8e66c325c3c44d983b2734062e5c71a"} Nov 03 21:48:51.176410 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.175987 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:48:51.227277 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.227201 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-envoy\") pod \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " Nov 03 21:48:51.227469 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.227387 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-podinfo\") pod \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " Nov 03 21:48:51.227469 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.227442 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-token\" (UniqueName: \"kubernetes.io/projected/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-token\") pod \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " Nov 03 21:48:51.227904 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.227490 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-workload-socket\") pod \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " Nov 03 21:48:51.227904 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.227548 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-credential-socket\") pod \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " Nov 03 21:48:51.227904 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.227623 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-workload-certs\") pod \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " Nov 03 21:48:51.227904 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.227700 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istiod-ca-cert\") pod \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " Nov 03 21:48:51.227904 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.227735 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-kube-api-access-xkpdc\" (UniqueName: \"kubernetes.io/projected/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-kube-api-access-xkpdc\") pod \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " Nov 03 21:48:51.227904 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.227839 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-data\") pod \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\" (UID: \"4b3eac22-c52b-48e7-bc7e-eddebf306cb5\") " Nov 03 21:48:51.228824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.228378 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-credential-socket" pod "4b3eac22-c52b-48e7-bc7e-eddebf306cb5" (UID: "4b3eac22-c52b-48e7-bc7e-eddebf306cb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.228824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.228583 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.228824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.228685 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-data" pod "4b3eac22-c52b-48e7-bc7e-eddebf306cb5" (UID: "4b3eac22-c52b-48e7-bc7e-eddebf306cb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.229059 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.228982 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istiod-ca-cert" pod "4b3eac22-c52b-48e7-bc7e-eddebf306cb5" (UID: "4b3eac22-c52b-48e7-bc7e-eddebf306cb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:51.229199 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.229182 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-workload-certs" pod "4b3eac22-c52b-48e7-bc7e-eddebf306cb5" (UID: "4b3eac22-c52b-48e7-bc7e-eddebf306cb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.230006 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.229946 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-workload-socket" pod "4b3eac22-c52b-48e7-bc7e-eddebf306cb5" (UID: "4b3eac22-c52b-48e7-bc7e-eddebf306cb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.230919 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.230893 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-envoy" pod "4b3eac22-c52b-48e7-bc7e-eddebf306cb5" (UID: "4b3eac22-c52b-48e7-bc7e-eddebf306cb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.232812 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.232788 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-podinfo" pod "4b3eac22-c52b-48e7-bc7e-eddebf306cb5" (UID: "4b3eac22-c52b-48e7-bc7e-eddebf306cb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:51.236476 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.236402 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-kube-api-access-xkpdc" pod "4b3eac22-c52b-48e7-bc7e-eddebf306cb5" (UID: "4b3eac22-c52b-48e7-bc7e-eddebf306cb5"). InnerVolumeSpecName "kube-api-access-xkpdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:51.238545 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.238521 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-token" pod "4b3eac22-c52b-48e7-bc7e-eddebf306cb5" (UID: "4b3eac22-c52b-48e7-bc7e-eddebf306cb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:51.245859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.245497 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:48:51.258191 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.258158 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:48:51.267002 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.266973 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:48:51.330272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.329643 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367832c5-176f-4416-afd0-5a1bdb2b7570-kube-api-access-w794m\" (UniqueName: \"kubernetes.io/projected/367832c5-176f-4416-afd0-5a1bdb2b7570-kube-api-access-w794m\") pod \"367832c5-176f-4416-afd0-5a1bdb2b7570\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " Nov 03 21:48:51.330272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.329723 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-credential-socket\") pod \"6ed859d9-82ee-46b1-9e96-776b7917656d\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " Nov 03 21:48:51.330272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.329768 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49882dd3-115a-4449-b98d-9773358608bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49882dd3-115a-4449-b98d-9773358608bc-istiod-ca-cert\") pod \"49882dd3-115a-4449-b98d-9773358608bc\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " Nov 03 21:48:51.330272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.329805 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49882dd3-115a-4449-b98d-9773358608bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49882dd3-115a-4449-b98d-9773358608bc-istio-podinfo\") pod \"49882dd3-115a-4449-b98d-9773358608bc\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " Nov 03 21:48:51.330272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.329842 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49882dd3-115a-4449-b98d-9773358608bc-istio-token\" (UniqueName: \"kubernetes.io/projected/49882dd3-115a-4449-b98d-9773358608bc-istio-token\") pod \"49882dd3-115a-4449-b98d-9773358608bc\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " Nov 03 21:48:51.330272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.329874 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-credential-socket\") pod \"49882dd3-115a-4449-b98d-9773358608bc\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " Nov 03 21:48:51.330272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.329921 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-envoy\") pod \"6ed859d9-82ee-46b1-9e96-776b7917656d\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " Nov 03 21:48:51.330272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.329952 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-podinfo\") pod \"6ed859d9-82ee-46b1-9e96-776b7917656d\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " Nov 03 21:48:51.330272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.329985 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-workload-certs\") pod \"49882dd3-115a-4449-b98d-9773358608bc\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " Nov 03 21:48:51.330272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.330020 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-data\") pod \"367832c5-176f-4416-afd0-5a1bdb2b7570\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " Nov 03 21:48:51.330957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.330394 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-credential-socket" pod "49882dd3-115a-4449-b98d-9773358608bc" (UID: "49882dd3-115a-4449-b98d-9773358608bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.330957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.330417 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-credential-socket" pod "6ed859d9-82ee-46b1-9e96-776b7917656d" (UID: "6ed859d9-82ee-46b1-9e96-776b7917656d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.330957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.330492 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49882dd3-115a-4449-b98d-9773358608bc-istiod-ca-cert" pod "49882dd3-115a-4449-b98d-9773358608bc" (UID: "49882dd3-115a-4449-b98d-9773358608bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:51.330957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.330547 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-token\" (UniqueName: \"kubernetes.io/projected/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-token\") pod \"6ed859d9-82ee-46b1-9e96-776b7917656d\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " Nov 03 21:48:51.330957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.330653 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-workload-certs" pod "49882dd3-115a-4449-b98d-9773358608bc" (UID: "49882dd3-115a-4449-b98d-9773358608bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.330957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.330731 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-data\") pod \"6ed859d9-82ee-46b1-9e96-776b7917656d\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " Nov 03 21:48:51.330957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.330773 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-workload-certs\") pod \"367832c5-176f-4416-afd0-5a1bdb2b7570\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " Nov 03 21:48:51.330957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.330823 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-workload-socket\") pod \"49882dd3-115a-4449-b98d-9773358608bc\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " Nov 03 21:48:51.330957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.330884 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-workload-socket\") pod \"6ed859d9-82ee-46b1-9e96-776b7917656d\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " Nov 03 21:48:51.330957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.330935 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-envoy\") pod \"367832c5-176f-4416-afd0-5a1bdb2b7570\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " Nov 03 21:48:51.331464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.330974 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-workload-socket\") pod \"367832c5-176f-4416-afd0-5a1bdb2b7570\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " Nov 03 21:48:51.331464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.330993 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-data" pod "6ed859d9-82ee-46b1-9e96-776b7917656d" (UID: "6ed859d9-82ee-46b1-9e96-776b7917656d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.331464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.331000 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-credential-socket\") pod \"367832c5-176f-4416-afd0-5a1bdb2b7570\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " Nov 03 21:48:51.331464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.331053 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-workload-certs" pod "367832c5-176f-4416-afd0-5a1bdb2b7570" (UID: "367832c5-176f-4416-afd0-5a1bdb2b7570"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.331464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.331141 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-workload-certs\") pod \"6ed859d9-82ee-46b1-9e96-776b7917656d\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " Nov 03 21:48:51.331464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.331209 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-token\" (UniqueName: \"kubernetes.io/projected/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-token\") pod \"367832c5-176f-4416-afd0-5a1bdb2b7570\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " Nov 03 21:48:51.331464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.331221 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-credential-socket" pod "367832c5-176f-4416-afd0-5a1bdb2b7570" (UID: "367832c5-176f-4416-afd0-5a1bdb2b7570"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.331464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.331244 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ed859d9-82ee-46b1-9e96-776b7917656d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed859d9-82ee-46b1-9e96-776b7917656d-istiod-ca-cert\") pod \"6ed859d9-82ee-46b1-9e96-776b7917656d\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " Nov 03 21:48:51.331464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.331259 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-workload-socket" pod "6ed859d9-82ee-46b1-9e96-776b7917656d" (UID: "6ed859d9-82ee-46b1-9e96-776b7917656d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.331464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.331293 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-istio-envoy\") pod \"49882dd3-115a-4449-b98d-9773358608bc\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " Nov 03 21:48:51.331464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.331330 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/367832c5-176f-4416-afd0-5a1bdb2b7570-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367832c5-176f-4416-afd0-5a1bdb2b7570-istiod-ca-cert\") pod \"367832c5-176f-4416-afd0-5a1bdb2b7570\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " Nov 03 21:48:51.331464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.331362 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed859d9-82ee-46b1-9e96-776b7917656d-kube-api-access-r8s24\" (UniqueName: \"kubernetes.io/projected/6ed859d9-82ee-46b1-9e96-776b7917656d-kube-api-access-r8s24\") pod \"6ed859d9-82ee-46b1-9e96-776b7917656d\" (UID: \"6ed859d9-82ee-46b1-9e96-776b7917656d\") " Nov 03 21:48:51.331464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.331389 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-podinfo\") pod \"367832c5-176f-4416-afd0-5a1bdb2b7570\" (UID: \"367832c5-176f-4416-afd0-5a1bdb2b7570\") " Nov 03 21:48:51.331464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.331419 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49882dd3-115a-4449-b98d-9773358608bc-kube-api-access-s4mv9\" (UniqueName: \"kubernetes.io/projected/49882dd3-115a-4449-b98d-9773358608bc-kube-api-access-s4mv9\") pod \"49882dd3-115a-4449-b98d-9773358608bc\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " Nov 03 21:48:51.331464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.331449 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-istio-data\") pod \"49882dd3-115a-4449-b98d-9773358608bc\" (UID: \"49882dd3-115a-4449-b98d-9773358608bc\") " Nov 03 21:48:51.332223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.331488 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-workload-socket" pod "49882dd3-115a-4449-b98d-9773358608bc" (UID: "49882dd3-115a-4449-b98d-9773358608bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.332223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.331507 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed859d9-82ee-46b1-9e96-776b7917656d-istiod-ca-cert" pod "6ed859d9-82ee-46b1-9e96-776b7917656d" (UID: "6ed859d9-82ee-46b1-9e96-776b7917656d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:51.332223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332042 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed859d9-82ee-46b1-9e96-776b7917656d-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.332223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332065 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.332223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332081 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.332223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332094 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.332223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332108 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49882dd3-115a-4449-b98d-9773358608bc-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.332223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332122 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.332223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332137 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.332223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332151 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.332223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332164 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.332223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332176 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.332223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332190 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.332223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332203 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.332223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332216 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.332223 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332229 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.333021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332242 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.333021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332258 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.333021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332271 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.333021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332285 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkpdc\" (UniqueName: \"kubernetes.io/projected/4b3eac22-c52b-48e7-bc7e-eddebf306cb5-kube-api-access-xkpdc\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.333021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332501 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-istio-data" pod "49882dd3-115a-4449-b98d-9773358608bc" (UID: "49882dd3-115a-4449-b98d-9773358608bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.333021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.332740 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-workload-socket" pod "367832c5-176f-4416-afd0-5a1bdb2b7570" (UID: "367832c5-176f-4416-afd0-5a1bdb2b7570"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.334781 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.334249 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-workload-certs" pod "6ed859d9-82ee-46b1-9e96-776b7917656d" (UID: "6ed859d9-82ee-46b1-9e96-776b7917656d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.334781 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.334266 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-data" pod "367832c5-176f-4416-afd0-5a1bdb2b7570" (UID: "367832c5-176f-4416-afd0-5a1bdb2b7570"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.334781 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.334705 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/367832c5-176f-4416-afd0-5a1bdb2b7570-istiod-ca-cert" pod "367832c5-176f-4416-afd0-5a1bdb2b7570" (UID: "367832c5-176f-4416-afd0-5a1bdb2b7570"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:51.335424 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.335129 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-istio-envoy" pod "49882dd3-115a-4449-b98d-9773358608bc" (UID: "49882dd3-115a-4449-b98d-9773358608bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.335424 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.335321 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed859d9-82ee-46b1-9e96-776b7917656d-kube-api-access-r8s24" pod "6ed859d9-82ee-46b1-9e96-776b7917656d" (UID: "6ed859d9-82ee-46b1-9e96-776b7917656d"). InnerVolumeSpecName "kube-api-access-r8s24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:51.336984 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.336958 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49882dd3-115a-4449-b98d-9773358608bc-kube-api-access-s4mv9" pod "49882dd3-115a-4449-b98d-9773358608bc" (UID: "49882dd3-115a-4449-b98d-9773358608bc"). InnerVolumeSpecName "kube-api-access-s4mv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:51.337830 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.337807 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-podinfo" pod "367832c5-176f-4416-afd0-5a1bdb2b7570" (UID: "367832c5-176f-4416-afd0-5a1bdb2b7570"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:51.343784 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.343640 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-token" pod "6ed859d9-82ee-46b1-9e96-776b7917656d" (UID: "6ed859d9-82ee-46b1-9e96-776b7917656d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:51.344078 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.344051 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49882dd3-115a-4449-b98d-9773358608bc-istio-podinfo" pod "49882dd3-115a-4449-b98d-9773358608bc" (UID: "49882dd3-115a-4449-b98d-9773358608bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:51.347376 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.344807 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-podinfo" pod "6ed859d9-82ee-46b1-9e96-776b7917656d" (UID: "6ed859d9-82ee-46b1-9e96-776b7917656d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:48:51.347376 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.344932 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-token" pod "367832c5-176f-4416-afd0-5a1bdb2b7570" (UID: "367832c5-176f-4416-afd0-5a1bdb2b7570"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:51.347376 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.345635 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-envoy" pod "367832c5-176f-4416-afd0-5a1bdb2b7570" (UID: "367832c5-176f-4416-afd0-5a1bdb2b7570"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.347587 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.347467 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367832c5-176f-4416-afd0-5a1bdb2b7570-kube-api-access-w794m" pod "367832c5-176f-4416-afd0-5a1bdb2b7570" (UID: "367832c5-176f-4416-afd0-5a1bdb2b7570"). InnerVolumeSpecName "kube-api-access-w794m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:51.347587 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.347495 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49882dd3-115a-4449-b98d-9773358608bc-istio-token" pod "49882dd3-115a-4449-b98d-9773358608bc" (UID: "49882dd3-115a-4449-b98d-9773358608bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:51.349151 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.349114 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-envoy" pod "6ed859d9-82ee-46b1-9e96-776b7917656d" (UID: "6ed859d9-82ee-46b1-9e96-776b7917656d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:48:51.433374 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433282 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.433374 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433314 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.433374 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433326 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.433374 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433337 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367832c5-176f-4416-afd0-5a1bdb2b7570-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.433374 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433350 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8s24\" (UniqueName: \"kubernetes.io/projected/6ed859d9-82ee-46b1-9e96-776b7917656d-kube-api-access-r8s24\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.433374 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433362 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.433374 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433377 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4mv9\" (UniqueName: \"kubernetes.io/projected/49882dd3-115a-4449-b98d-9773358608bc-kube-api-access-s4mv9\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.433756 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433389 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49882dd3-115a-4449-b98d-9773358608bc-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.433756 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433401 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w794m\" (UniqueName: \"kubernetes.io/projected/367832c5-176f-4416-afd0-5a1bdb2b7570-kube-api-access-w794m\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.433756 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433412 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49882dd3-115a-4449-b98d-9773358608bc-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.433756 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433423 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49882dd3-115a-4449-b98d-9773358608bc-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.433756 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433434 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.433756 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433445 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.433756 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433455 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.433756 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433466 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed859d9-82ee-46b1-9e96-776b7917656d-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.433756 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433480 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:51.433756 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:51.433491 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367832c5-176f-4416-afd0-5a1bdb2b7570-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:52.128213 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.128177 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" Nov 03 21:48:52.128392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.128206 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr" event={"ID":"4b3eac22-c52b-48e7-bc7e-eddebf306cb5","Type":"ContainerDied","Data":"4a6c0b110599ec08ace65cd9a53a59cea78cdd05fad9050fceac221973c307e3"} Nov 03 21:48:52.128392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.128279 5041 scope.go:122] "RemoveContainer" containerID="6491ab9a2d3123365fcf19697178a8895552eae48eb6fb8dadef33e6f11ffff2" Nov 03 21:48:52.130757 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.130358 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" Nov 03 21:48:52.130757 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.130377 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn" event={"ID":"49882dd3-115a-4449-b98d-9773358608bc","Type":"ContainerDied","Data":"c23ea08d486ae9e0a6c42a1cace4450c03684af236d4a4fdf8c4cd82e9e81a23"} Nov 03 21:48:52.132946 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.132923 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" Nov 03 21:48:52.132946 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.132932 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs" event={"ID":"367832c5-176f-4416-afd0-5a1bdb2b7570","Type":"ContainerDied","Data":"55b5a9e1c0fe674eda0e4d014f7c8a388804773777fe5e804269b8f5b49701ce"} Nov 03 21:48:52.134631 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.134594 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" event={"ID":"6ed859d9-82ee-46b1-9e96-776b7917656d","Type":"ContainerDied","Data":"c4addd70e47f15bb9aa11aaff65f5aebcc2c478b4ba0f1757486e265656d556f"} Nov 03 21:48:52.134741 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.134723 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6" Nov 03 21:48:52.147030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.147011 5041 scope.go:122] "RemoveContainer" containerID="d817f1837d733f59ff5a94fc2062e5a0b8e66c325c3c44d983b2734062e5c71a" Nov 03 21:48:52.158146 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.158119 5041 scope.go:122] "RemoveContainer" containerID="ce79be379e644fad8ba3c9f4ce0b02c78ab6876da16be67adedb10dad3a219db" Nov 03 21:48:52.171436 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.171413 5041 scope.go:122] "RemoveContainer" containerID="26b7768609b866b137d560a15c95e5b6978bb2181ac1affd75a2f3183cae6afa" Nov 03 21:48:52.233957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.233922 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr"] Nov 03 21:48:52.247628 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.247571 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-lr4kr"] Nov 03 21:48:52.276985 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.276954 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6"] Nov 03 21:48:52.283995 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.283971 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-rc2p6"] Nov 03 21:48:52.313963 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.313915 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs"] Nov 03 21:48:52.319012 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.318987 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-54pcs"] Nov 03 21:48:52.351812 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.351774 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn"] Nov 03 21:48:52.356845 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.356820 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-27fdf4-openshift-default-67c598bf99-2nkrn"] Nov 03 21:48:52.622156 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.622121 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" path="/var/lib/kubelet/pods/367832c5-176f-4416-afd0-5a1bdb2b7570/volumes" Nov 03 21:48:52.622727 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.622704 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49882dd3-115a-4449-b98d-9773358608bc" path="/var/lib/kubelet/pods/49882dd3-115a-4449-b98d-9773358608bc/volumes" Nov 03 21:48:52.623214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.623194 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" path="/var/lib/kubelet/pods/4b3eac22-c52b-48e7-bc7e-eddebf306cb5/volumes" Nov 03 21:48:52.624026 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:52.624003 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ed859d9-82ee-46b1-9e96-776b7917656d" path="/var/lib/kubelet/pods/6ed859d9-82ee-46b1-9e96-776b7917656d/volumes" Nov 03 21:48:53.787898 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:53.787857 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272"] Nov 03 21:48:54.962345 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.962307 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5"] Nov 03 21:48:54.962879 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.962864 5041 cpu_manager.go:414] "containerMap: removing container" podUID="28004d42-99bd-44a3-9e2e-94c4189965f0" containerName="revoke-keys" Nov 03 21:48:54.962939 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.962880 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28004d42-99bd-44a3-9e2e-94c4189965f0" containerName="revoke-keys" Nov 03 21:48:54.962939 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.962891 5041 cpu_manager.go:414] "containerMap: removing container" podUID="6ed859d9-82ee-46b1-9e96-776b7917656d" containerName="istio-proxy" Nov 03 21:48:54.962939 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.962897 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed859d9-82ee-46b1-9e96-776b7917656d" containerName="istio-proxy" Nov 03 21:48:54.962939 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.962917 5041 cpu_manager.go:414] "containerMap: removing container" podUID="668e0b57-1f4e-441f-8d8b-fddc8524fcf6" containerName="istio-proxy" Nov 03 21:48:54.962939 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.962924 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668e0b57-1f4e-441f-8d8b-fddc8524fcf6" containerName="istio-proxy" Nov 03 21:48:54.962939 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.962939 5041 cpu_manager.go:414] "containerMap: removing container" podUID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" containerName="istio-proxy" Nov 03 21:48:54.963111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.962944 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" containerName="istio-proxy" Nov 03 21:48:54.963111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.962952 5041 cpu_manager.go:414] "containerMap: removing container" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" Nov 03 21:48:54.963111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.962956 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" Nov 03 21:48:54.963111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.962971 5041 cpu_manager.go:414] "containerMap: removing container" podUID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" containerName="istio-proxy" Nov 03 21:48:54.963111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.962976 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" containerName="istio-proxy" Nov 03 21:48:54.963111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.962986 5041 cpu_manager.go:414] "containerMap: removing container" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" Nov 03 21:48:54.963111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.962991 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" Nov 03 21:48:54.963111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.963082 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" containerName="istio-proxy" Nov 03 21:48:54.963111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.963091 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="367832c5-176f-4416-afd0-5a1bdb2b7570" containerName="istio-proxy" Nov 03 21:48:54.963111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.963101 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b3eac22-c52b-48e7-bc7e-eddebf306cb5" containerName="istio-proxy" Nov 03 21:48:54.963111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.963108 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="49882dd3-115a-4449-b98d-9773358608bc" containerName="istio-proxy" Nov 03 21:48:54.963111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.963114 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="28004d42-99bd-44a3-9e2e-94c4189965f0" containerName="revoke-keys" Nov 03 21:48:54.963466 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.963120 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed859d9-82ee-46b1-9e96-776b7917656d" containerName="istio-proxy" Nov 03 21:48:54.963466 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.963127 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="668e0b57-1f4e-441f-8d8b-fddc8524fcf6" containerName="istio-proxy" Nov 03 21:48:54.968839 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.968814 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" Nov 03 21:48:54.986707 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.986654 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5"] Nov 03 21:48:54.993547 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.993514 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:48:54.993706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:54.993578 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:48:55.090965 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:55.090924 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5150c7e7-763a-4b01-b493-d4046712ff6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65bf3fab-9bkz5\" (UID: \"5150c7e7-763a-4b01-b493-d4046712ff6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" Nov 03 21:48:55.091160 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:55.091053 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-586f2\" (UniqueName: \"kubernetes.io/projected/5150c7e7-763a-4b01-b493-d4046712ff6d-kube-api-access-586f2\") pod \"maas-api-revoke-sub-65bf3fab-9bkz5\" (UID: \"5150c7e7-763a-4b01-b493-d4046712ff6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" Nov 03 21:48:55.091211 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:55.091195 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5150c7e7-763a-4b01-b493-d4046712ff6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65bf3fab-9bkz5\" (UID: \"5150c7e7-763a-4b01-b493-d4046712ff6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" Nov 03 21:48:55.192357 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:55.192325 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5150c7e7-763a-4b01-b493-d4046712ff6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65bf3fab-9bkz5\" (UID: \"5150c7e7-763a-4b01-b493-d4046712ff6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" Nov 03 21:48:55.192555 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:55.192386 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5150c7e7-763a-4b01-b493-d4046712ff6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65bf3fab-9bkz5\" (UID: \"5150c7e7-763a-4b01-b493-d4046712ff6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" Nov 03 21:48:55.192555 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:55.192462 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-586f2\" (UniqueName: \"kubernetes.io/projected/5150c7e7-763a-4b01-b493-d4046712ff6d-kube-api-access-586f2\") pod \"maas-api-revoke-sub-65bf3fab-9bkz5\" (UID: \"5150c7e7-763a-4b01-b493-d4046712ff6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" Nov 03 21:48:55.193160 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:55.193131 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5150c7e7-763a-4b01-b493-d4046712ff6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65bf3fab-9bkz5\" (UID: \"5150c7e7-763a-4b01-b493-d4046712ff6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" Nov 03 21:48:55.194876 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:55.194849 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5150c7e7-763a-4b01-b493-d4046712ff6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65bf3fab-9bkz5\" (UID: \"5150c7e7-763a-4b01-b493-d4046712ff6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" Nov 03 21:48:55.212946 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:55.212875 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-586f2\" (UniqueName: \"kubernetes.io/projected/5150c7e7-763a-4b01-b493-d4046712ff6d-kube-api-access-586f2\") pod \"maas-api-revoke-sub-65bf3fab-9bkz5\" (UID: \"5150c7e7-763a-4b01-b493-d4046712ff6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" Nov 03 21:48:55.289170 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:55.289136 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" Nov 03 21:48:55.590155 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:48:55.590080 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5150c7e7_763a_4b01_b493_d4046712ff6d.slice/crio-781cf20a3f74b36c76cbb0f82711a24b96fe19a3e13f8ff6c0420b9b685988dd WatchSource:0}: Error finding container 781cf20a3f74b36c76cbb0f82711a24b96fe19a3e13f8ff6c0420b9b685988dd: Status 404 returned error can't find the container with id 781cf20a3f74b36c76cbb0f82711a24b96fe19a3e13f8ff6c0420b9b685988dd Nov 03 21:48:55.592631 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:55.592600 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5"] Nov 03 21:48:56.177140 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:56.177045 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" event={"ID":"5150c7e7-763a-4b01-b493-d4046712ff6d","Type":"ContainerStarted","Data":"e2b9363c30ce714ed7e6519b6a11be66ee44292afb63ad19b7c01bd971897622"} Nov 03 21:48:56.177140 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:56.177115 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" event={"ID":"5150c7e7-763a-4b01-b493-d4046712ff6d","Type":"ContainerStarted","Data":"781cf20a3f74b36c76cbb0f82711a24b96fe19a3e13f8ff6c0420b9b685988dd"} Nov 03 21:48:56.221113 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:56.221052 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" podStartSLOduration=2.221034585 podStartE2EDuration="2.221034585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:56.2176033 +0000 UTC m=+1151.724831594" watchObservedRunningTime="2026-11-03 21:48:56.221034585 +0000 UTC m=+1151.728262864" Nov 03 21:48:57.034743 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:57.034638 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w"] Nov 03 21:48:57.047934 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:57.047899 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79ae8406-bmg9w"] Nov 03 21:48:57.187414 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:57.187370 5041 generic.go:356] "Generic (PLEG): container finished" podID="5150c7e7-763a-4b01-b493-d4046712ff6d" containerID="e2b9363c30ce714ed7e6519b6a11be66ee44292afb63ad19b7c01bd971897622" exitCode=0 Nov 03 21:48:57.187830 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:57.187455 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" event={"ID":"5150c7e7-763a-4b01-b493-d4046712ff6d","Type":"ContainerDied","Data":"e2b9363c30ce714ed7e6519b6a11be66ee44292afb63ad19b7c01bd971897622"} Nov 03 21:48:58.343262 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:58.343236 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" Nov 03 21:48:58.431236 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:58.431199 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5150c7e7-763a-4b01-b493-d4046712ff6d-kube-api-access-586f2\" (UniqueName: \"kubernetes.io/projected/5150c7e7-763a-4b01-b493-d4046712ff6d-kube-api-access-586f2\") pod \"5150c7e7-763a-4b01-b493-d4046712ff6d\" (UID: \"5150c7e7-763a-4b01-b493-d4046712ff6d\") " Nov 03 21:48:58.431409 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:58.431344 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5150c7e7-763a-4b01-b493-d4046712ff6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5150c7e7-763a-4b01-b493-d4046712ff6d-maas-api-openshift-ca\") pod \"5150c7e7-763a-4b01-b493-d4046712ff6d\" (UID: \"5150c7e7-763a-4b01-b493-d4046712ff6d\") " Nov 03 21:48:58.431531 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:58.431511 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5150c7e7-763a-4b01-b493-d4046712ff6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5150c7e7-763a-4b01-b493-d4046712ff6d-maas-api-xks-ca\") pod \"5150c7e7-763a-4b01-b493-d4046712ff6d\" (UID: \"5150c7e7-763a-4b01-b493-d4046712ff6d\") " Nov 03 21:48:58.431731 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:58.431708 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5150c7e7-763a-4b01-b493-d4046712ff6d-maas-api-openshift-ca" pod "5150c7e7-763a-4b01-b493-d4046712ff6d" (UID: "5150c7e7-763a-4b01-b493-d4046712ff6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:48:58.431930 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:58.431916 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5150c7e7-763a-4b01-b493-d4046712ff6d-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:58.433570 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:58.433542 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5150c7e7-763a-4b01-b493-d4046712ff6d-kube-api-access-586f2" pod "5150c7e7-763a-4b01-b493-d4046712ff6d" (UID: "5150c7e7-763a-4b01-b493-d4046712ff6d"). InnerVolumeSpecName "kube-api-access-586f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:48:58.433570 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:58.433551 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5150c7e7-763a-4b01-b493-d4046712ff6d-maas-api-xks-ca" pod "5150c7e7-763a-4b01-b493-d4046712ff6d" (UID: "5150c7e7-763a-4b01-b493-d4046712ff6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:48:58.532745 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:58.532713 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5150c7e7-763a-4b01-b493-d4046712ff6d-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:58.532745 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:58.532742 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-586f2\" (UniqueName: \"kubernetes.io/projected/5150c7e7-763a-4b01-b493-d4046712ff6d-kube-api-access-586f2\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:48:58.622017 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:58.621974 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81" path="/var/lib/kubelet/pods/ce4e40f3-c0f0-4be2-a5e6-2d94850e0a81/volumes" Nov 03 21:48:58.788692 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:58.788571 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" podUID="20b45af0-efad-4f42-a998-784bf9f39be9" containerName="payload-pre-processing" containerID="cri-o://11d77de13d464a4bbda88b9ec3a2b66b34115d9c427383f1f05aca24c58841fb" gracePeriod=25 Nov 03 21:48:59.205559 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:59.205508 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" Nov 03 21:48:59.205559 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:59.205513 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5" event={"ID":"5150c7e7-763a-4b01-b493-d4046712ff6d","Type":"ContainerDied","Data":"781cf20a3f74b36c76cbb0f82711a24b96fe19a3e13f8ff6c0420b9b685988dd"} Nov 03 21:48:59.205883 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:48:59.205594 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="781cf20a3f74b36c76cbb0f82711a24b96fe19a3e13f8ff6c0420b9b685988dd" Nov 03 21:49:02.035319 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:02.035279 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj"] Nov 03 21:49:02.049096 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:02.049061 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh"] Nov 03 21:49:02.053697 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:02.053648 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d3f6fe1-flwcj"] Nov 03 21:49:02.058174 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:02.058149 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1e57fd5-fssrh"] Nov 03 21:49:02.622064 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:02.622027 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7011cf-b114-4f2b-9ff5-f2072021c2b2" path="/var/lib/kubelet/pods/6b7011cf-b114-4f2b-9ff5-f2072021c2b2/volumes" Nov 03 21:49:02.622489 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:02.622475 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de348e53-4616-4e56-9fde-954cff5e5f89" path="/var/lib/kubelet/pods/de348e53-4616-4e56-9fde-954cff5e5f89/volumes" Nov 03 21:49:03.152325 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.152285 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" containerID="cri-o://ca81f569e3516dba457a5eacb9e28a971536e990d82c4510b41a62077f3025cf" gracePeriod=45 Nov 03 21:49:03.162869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.162828 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" containerID="cri-o://ab205213a85bcef421c895b93b41447eb7df71dbe0fff7d15d99736e154adc51" gracePeriod=45 Nov 03 21:49:03.456361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.456325 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:49:03.481568 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.481535 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:49:03.493622 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.493583 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-home\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-home\") pod \"0208e182-3db8-43cc-a945-9cea424a6796\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " Nov 03 21:49:03.493834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.493654 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-home\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-home\") pod \"176dd3a2-8222-4882-97d7-27e51c5cd526\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " Nov 03 21:49:03.493834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.493747 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-tmp-dir\") pod \"176dd3a2-8222-4882-97d7-27e51c5cd526\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " Nov 03 21:49:03.493834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.493783 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176dd3a2-8222-4882-97d7-27e51c5cd526-kube-api-access-4t9cm\" (UniqueName: \"kubernetes.io/projected/176dd3a2-8222-4882-97d7-27e51c5cd526-kube-api-access-4t9cm\") pod \"176dd3a2-8222-4882-97d7-27e51c5cd526\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " Nov 03 21:49:03.493834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.493828 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0208e182-3db8-43cc-a945-9cea424a6796-tls-certs\" (UniqueName: \"kubernetes.io/secret/0208e182-3db8-43cc-a945-9cea424a6796-tls-certs\") pod \"0208e182-3db8-43cc-a945-9cea424a6796\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " Nov 03 21:49:03.494054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.493857 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0208e182-3db8-43cc-a945-9cea424a6796-kube-api-access-7nf95\" (UniqueName: \"kubernetes.io/projected/0208e182-3db8-43cc-a945-9cea424a6796-kube-api-access-7nf95\") pod \"0208e182-3db8-43cc-a945-9cea424a6796\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " Nov 03 21:49:03.494054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.493895 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-home" pod "0208e182-3db8-43cc-a945-9cea424a6796" (UID: "0208e182-3db8-43cc-a945-9cea424a6796"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:03.494054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.493906 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-model-cache\") pod \"0208e182-3db8-43cc-a945-9cea424a6796\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " Nov 03 21:49:03.494054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.493946 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-home" pod "176dd3a2-8222-4882-97d7-27e51c5cd526" (UID: "176dd3a2-8222-4882-97d7-27e51c5cd526"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:03.494054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.493992 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-tmp-dir" pod "176dd3a2-8222-4882-97d7-27e51c5cd526" (UID: "176dd3a2-8222-4882-97d7-27e51c5cd526"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:03.494054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.494006 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-dshm\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-dshm\") pod \"176dd3a2-8222-4882-97d7-27e51c5cd526\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " Nov 03 21:49:03.494335 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.494060 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-dshm\") pod \"0208e182-3db8-43cc-a945-9cea424a6796\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " Nov 03 21:49:03.494335 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.494130 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-tmp-dir\") pod \"0208e182-3db8-43cc-a945-9cea424a6796\" (UID: \"0208e182-3db8-43cc-a945-9cea424a6796\") " Nov 03 21:49:03.494335 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.494132 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-model-cache" pod "0208e182-3db8-43cc-a945-9cea424a6796" (UID: "0208e182-3db8-43cc-a945-9cea424a6796"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:03.494335 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.494162 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-model-cache\") pod \"176dd3a2-8222-4882-97d7-27e51c5cd526\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " Nov 03 21:49:03.494335 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.494214 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176dd3a2-8222-4882-97d7-27e51c5cd526-tls-certs\" (UniqueName: \"kubernetes.io/secret/176dd3a2-8222-4882-97d7-27e51c5cd526-tls-certs\") pod \"176dd3a2-8222-4882-97d7-27e51c5cd526\" (UID: \"176dd3a2-8222-4882-97d7-27e51c5cd526\") " Nov 03 21:49:03.494521 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.494494 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-tmp-dir" pod "0208e182-3db8-43cc-a945-9cea424a6796" (UID: "0208e182-3db8-43cc-a945-9cea424a6796"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:03.494783 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.494728 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-model-cache" pod "176dd3a2-8222-4882-97d7-27e51c5cd526" (UID: "176dd3a2-8222-4882-97d7-27e51c5cd526"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:03.494783 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.494752 5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-tmp-dir\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:03.494783 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.494775 5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-home\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:03.496055 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.494789 5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-home\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:03.496055 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.494802 5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-tmp-dir\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:03.496055 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.494813 5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-model-cache\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:03.498502 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.498272 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176dd3a2-8222-4882-97d7-27e51c5cd526-tls-certs" pod "176dd3a2-8222-4882-97d7-27e51c5cd526" (UID: "176dd3a2-8222-4882-97d7-27e51c5cd526"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:49:03.498502 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.498470 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0208e182-3db8-43cc-a945-9cea424a6796-kube-api-access-7nf95" pod "0208e182-3db8-43cc-a945-9cea424a6796" (UID: "0208e182-3db8-43cc-a945-9cea424a6796"). InnerVolumeSpecName "kube-api-access-7nf95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:03.498656 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.498556 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176dd3a2-8222-4882-97d7-27e51c5cd526-kube-api-access-4t9cm" pod "176dd3a2-8222-4882-97d7-27e51c5cd526" (UID: "176dd3a2-8222-4882-97d7-27e51c5cd526"). InnerVolumeSpecName "kube-api-access-4t9cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:03.498784 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.498750 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0208e182-3db8-43cc-a945-9cea424a6796-tls-certs" pod "0208e182-3db8-43cc-a945-9cea424a6796" (UID: "0208e182-3db8-43cc-a945-9cea424a6796"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:49:03.498848 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.498783 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-dshm" pod "176dd3a2-8222-4882-97d7-27e51c5cd526" (UID: "176dd3a2-8222-4882-97d7-27e51c5cd526"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:03.499993 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.499973 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-dshm" pod "0208e182-3db8-43cc-a945-9cea424a6796" (UID: "0208e182-3db8-43cc-a945-9cea424a6796"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:03.596392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.596362 5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-model-cache\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:03.596392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.596390 5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176dd3a2-8222-4882-97d7-27e51c5cd526-tls-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:03.596392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.596400 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9cm\" (UniqueName: \"kubernetes.io/projected/176dd3a2-8222-4882-97d7-27e51c5cd526-kube-api-access-4t9cm\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:03.596633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.596413 5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0208e182-3db8-43cc-a945-9cea424a6796-tls-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:03.596633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.596424 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nf95\" (UniqueName: \"kubernetes.io/projected/0208e182-3db8-43cc-a945-9cea424a6796-kube-api-access-7nf95\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:03.596633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.596438 5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176dd3a2-8222-4882-97d7-27e51c5cd526-dshm\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:03.596633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:03.596450 5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0208e182-3db8-43cc-a945-9cea424a6796-dshm\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:04.251169 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.251130 5041 generic.go:356] "Generic (PLEG): container finished" podID="0208e182-3db8-43cc-a945-9cea424a6796" containerID="ca81f569e3516dba457a5eacb9e28a971536e990d82c4510b41a62077f3025cf" exitCode=0 Nov 03 21:49:04.251638 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.251196 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" Nov 03 21:49:04.251638 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.251212 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" event={"ID":"0208e182-3db8-43cc-a945-9cea424a6796","Type":"ContainerDied","Data":"ca81f569e3516dba457a5eacb9e28a971536e990d82c4510b41a62077f3025cf"} Nov 03 21:49:04.251638 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.251292 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz" event={"ID":"0208e182-3db8-43cc-a945-9cea424a6796","Type":"ContainerDied","Data":"cf69f419533b3386337d70c999bfd2dca9e18049b139f8c4e718d8009b837f65"} Nov 03 21:49:04.251638 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.251340 5041 scope.go:122] "RemoveContainer" containerID="ca81f569e3516dba457a5eacb9e28a971536e990d82c4510b41a62077f3025cf" Nov 03 21:49:04.253215 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.253194 5041 generic.go:356] "Generic (PLEG): container finished" podID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerID="ab205213a85bcef421c895b93b41447eb7df71dbe0fff7d15d99736e154adc51" exitCode=0 Nov 03 21:49:04.253335 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.253240 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" Nov 03 21:49:04.253335 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.253278 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" event={"ID":"176dd3a2-8222-4882-97d7-27e51c5cd526","Type":"ContainerDied","Data":"ab205213a85bcef421c895b93b41447eb7df71dbe0fff7d15d99736e154adc51"} Nov 03 21:49:04.253335 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.253326 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5" event={"ID":"176dd3a2-8222-4882-97d7-27e51c5cd526","Type":"ContainerDied","Data":"29a0a99d38b1c01aa4cfb097418b7f9265607ab145079f97e4c451bb3c7924c9"} Nov 03 21:49:04.264601 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.264578 5041 scope.go:122] "RemoveContainer" containerID="ca81f569e3516dba457a5eacb9e28a971536e990d82c4510b41a62077f3025cf" Nov 03 21:49:04.264987 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:49:04.264959 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca81f569e3516dba457a5eacb9e28a971536e990d82c4510b41a62077f3025cf\": container with ID starting with ca81f569e3516dba457a5eacb9e28a971536e990d82c4510b41a62077f3025cf not found: ID does not exist" containerID="ca81f569e3516dba457a5eacb9e28a971536e990d82c4510b41a62077f3025cf" Nov 03 21:49:04.265076 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.264998 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca81f569e3516dba457a5eacb9e28a971536e990d82c4510b41a62077f3025cf"} err="failed to get container status \"ca81f569e3516dba457a5eacb9e28a971536e990d82c4510b41a62077f3025cf\": rpc error: code = NotFound desc = could not find container \"ca81f569e3516dba457a5eacb9e28a971536e990d82c4510b41a62077f3025cf\": container with ID starting with ca81f569e3516dba457a5eacb9e28a971536e990d82c4510b41a62077f3025cf not found: ID does not exist" Nov 03 21:49:04.265076 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.265021 5041 scope.go:122] "RemoveContainer" containerID="ab205213a85bcef421c895b93b41447eb7df71dbe0fff7d15d99736e154adc51" Nov 03 21:49:04.275704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.275683 5041 scope.go:122] "RemoveContainer" containerID="ab205213a85bcef421c895b93b41447eb7df71dbe0fff7d15d99736e154adc51" Nov 03 21:49:04.276323 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:49:04.276298 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab205213a85bcef421c895b93b41447eb7df71dbe0fff7d15d99736e154adc51\": container with ID starting with ab205213a85bcef421c895b93b41447eb7df71dbe0fff7d15d99736e154adc51 not found: ID does not exist" containerID="ab205213a85bcef421c895b93b41447eb7df71dbe0fff7d15d99736e154adc51" Nov 03 21:49:04.276396 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.276329 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab205213a85bcef421c895b93b41447eb7df71dbe0fff7d15d99736e154adc51"} err="failed to get container status \"ab205213a85bcef421c895b93b41447eb7df71dbe0fff7d15d99736e154adc51\": rpc error: code = NotFound desc = could not find container \"ab205213a85bcef421c895b93b41447eb7df71dbe0fff7d15d99736e154adc51\": container with ID starting with ab205213a85bcef421c895b93b41447eb7df71dbe0fff7d15d99736e154adc51 not found: ID does not exist" Nov 03 21:49:04.308649 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.308617 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz"] Nov 03 21:49:04.322141 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.322113 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-517149/premium-w6-517149-kserve-69fb88479-zwdcz"] Nov 03 21:49:04.348085 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.348057 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5"] Nov 03 21:49:04.356179 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.356152 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-517149/simulator-w6-517149-kserve-56b657fc78-xscq5"] Nov 03 21:49:04.623028 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.622999 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0208e182-3db8-43cc-a945-9cea424a6796" path="/var/lib/kubelet/pods/0208e182-3db8-43cc-a945-9cea424a6796/volumes" Nov 03 21:49:04.623403 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:04.623389 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" path="/var/lib/kubelet/pods/176dd3a2-8222-4882-97d7-27e51c5cd526/volumes" Nov 03 21:49:07.161386 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:07.161346 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" podUID="20b45af0-efad-4f42-a998-784bf9f39be9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:49:09.028523 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:09.028486 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp"] Nov 03 21:49:09.041684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:09.041633 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5498eaf4-5qtqp"] Nov 03 21:49:10.070748 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:10.070706 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8"] Nov 03 21:49:10.101186 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:10.101145 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288fc996-b9dm8"] Nov 03 21:49:10.637405 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:10.637362 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca4cd03-2b97-4cc8-9306-ea44a53e0668" path="/var/lib/kubelet/pods/8ca4cd03-2b97-4cc8-9306-ea44a53e0668/volumes" Nov 03 21:49:10.638331 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:10.638306 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ab6e34-3152-4ece-a98e-4fc7f59ec674" path="/var/lib/kubelet/pods/b1ab6e34-3152-4ece-a98e-4fc7f59ec674/volumes" Nov 03 21:49:10.785480 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:49:10.785445 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:49:10.876307 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:10.876271 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj"] Nov 03 21:49:10.890825 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:10.890693 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-517149-zpddj"] Nov 03 21:49:12.621275 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:12.621238 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28004d42-99bd-44a3-9e2e-94c4189965f0" path="/var/lib/kubelet/pods/28004d42-99bd-44a3-9e2e-94c4189965f0/volumes" Nov 03 21:49:17.162165 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:17.162126 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" podUID="20b45af0-efad-4f42-a998-784bf9f39be9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:49:17.413786 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:17.413646 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc"] Nov 03 21:49:17.414498 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:17.414433 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" podUID="b68bf331-de7c-4e66-9371-72984471e65e" containerName="istio-proxy" containerID="cri-o://8377c76c2c5611506b0db9607b8ecb564ec0325f748ef853b32baeb6f123852b" gracePeriod=30 Nov 03 21:49:17.433892 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:17.433851 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq"] Nov 03 21:49:17.434175 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:17.434147 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" podUID="9507b5d4-33e4-470f-9385-59164cb2e72a" containerName="istio-proxy" containerID="cri-o://d4a7661eb514e5931ceb633572b4ab35f215cd778792a6cc46fd057c387b92ce" gracePeriod=30 Nov 03 21:49:17.441136 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:17.441106 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv"] Nov 03 21:49:17.441369 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:17.441332 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" podUID="277d0d55-608c-451a-a704-0a80454055bc" containerName="istio-proxy" containerID="cri-o://bda53a79e99c30d00c517ce9375f3a9f4644f5469301dd2a173d3f70df0c83df" gracePeriod=30 Nov 03 21:49:19.069704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.069651 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" Nov 03 21:49:19.124994 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.117003 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b45af0-efad-4f42-a998-784bf9f39be9-kube-api-access-4bmkd\" (UniqueName: \"kubernetes.io/projected/20b45af0-efad-4f42-a998-784bf9f39be9-kube-api-access-4bmkd\") pod \"20b45af0-efad-4f42-a998-784bf9f39be9\" (UID: \"20b45af0-efad-4f42-a998-784bf9f39be9\") " Nov 03 21:49:19.124994 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.117206 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b45af0-efad-4f42-a998-784bf9f39be9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20b45af0-efad-4f42-a998-784bf9f39be9-plugins-config-volume\") pod \"20b45af0-efad-4f42-a998-784bf9f39be9\" (UID: \"20b45af0-efad-4f42-a998-784bf9f39be9\") " Nov 03 21:49:19.126111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.126052 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b45af0-efad-4f42-a998-784bf9f39be9-plugins-config-volume" pod "20b45af0-efad-4f42-a998-784bf9f39be9" (UID: "20b45af0-efad-4f42-a998-784bf9f39be9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:49:19.134908 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.134856 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b45af0-efad-4f42-a998-784bf9f39be9-kube-api-access-4bmkd" pod "20b45af0-efad-4f42-a998-784bf9f39be9" (UID: "20b45af0-efad-4f42-a998-784bf9f39be9"). InnerVolumeSpecName "kube-api-access-4bmkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:19.218329 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.218292 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bmkd\" (UniqueName: \"kubernetes.io/projected/20b45af0-efad-4f42-a998-784bf9f39be9-kube-api-access-4bmkd\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:19.218329 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.218324 5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20b45af0-efad-4f42-a998-784bf9f39be9-plugins-config-volume\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:19.510383 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.510347 5041 patch_prober.go:29] interesting pod/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:49:19.510582 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.510404 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" podUID="b68bf331-de7c-4e66-9371-72984471e65e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:49:19.568450 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.568413 5041 generic.go:356] "Generic (PLEG): container finished" podID="20b45af0-efad-4f42-a998-784bf9f39be9" containerID="11d77de13d464a4bbda88b9ec3a2b66b34115d9c427383f1f05aca24c58841fb" exitCode=0 Nov 03 21:49:19.568633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.568462 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" Nov 03 21:49:19.568633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.568488 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" event={"ID":"20b45af0-efad-4f42-a998-784bf9f39be9","Type":"ContainerDied","Data":"11d77de13d464a4bbda88b9ec3a2b66b34115d9c427383f1f05aca24c58841fb"} Nov 03 21:49:19.568633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.568548 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272" event={"ID":"20b45af0-efad-4f42-a998-784bf9f39be9","Type":"ContainerDied","Data":"d6138f637254794aa8dc7a99586171220106bf4db1ddcca161e79303ae12e766"} Nov 03 21:49:19.568633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.568572 5041 scope.go:122] "RemoveContainer" containerID="11d77de13d464a4bbda88b9ec3a2b66b34115d9c427383f1f05aca24c58841fb" Nov 03 21:49:19.579884 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.579860 5041 scope.go:122] "RemoveContainer" containerID="11d77de13d464a4bbda88b9ec3a2b66b34115d9c427383f1f05aca24c58841fb" Nov 03 21:49:19.580171 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:49:19.580149 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d77de13d464a4bbda88b9ec3a2b66b34115d9c427383f1f05aca24c58841fb\": container with ID starting with 11d77de13d464a4bbda88b9ec3a2b66b34115d9c427383f1f05aca24c58841fb not found: ID does not exist" containerID="11d77de13d464a4bbda88b9ec3a2b66b34115d9c427383f1f05aca24c58841fb" Nov 03 21:49:19.580212 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.580181 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d77de13d464a4bbda88b9ec3a2b66b34115d9c427383f1f05aca24c58841fb"} err="failed to get container status \"11d77de13d464a4bbda88b9ec3a2b66b34115d9c427383f1f05aca24c58841fb\": rpc error: code = NotFound desc = could not find container \"11d77de13d464a4bbda88b9ec3a2b66b34115d9c427383f1f05aca24c58841fb\": container with ID starting with 11d77de13d464a4bbda88b9ec3a2b66b34115d9c427383f1f05aca24c58841fb not found: ID does not exist" Nov 03 21:49:19.625502 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.625465 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272"] Nov 03 21:49:19.638999 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:19.638967 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-517149-6db48856c5-qx272"] Nov 03 21:49:20.622415 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:20.622372 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b45af0-efad-4f42-a998-784bf9f39be9" path="/var/lib/kubelet/pods/20b45af0-efad-4f42-a998-784bf9f39be9/volumes" Nov 03 21:49:21.029397 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:21.029363 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb"] Nov 03 21:49:21.042977 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:21.042941 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d86cc3ff-m2nqb"] Nov 03 21:49:22.071132 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.071089 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g"] Nov 03 21:49:22.084627 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.084593 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f317155d-jn47g"] Nov 03 21:49:22.377805 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.377707 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d575fdb56-8br4h"] Nov 03 21:49:22.378042 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.377986 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6d575fdb56-8br4h" podUID="d82b63ac-65d4-4e30-8b51-d308cb3d335d" containerName="manager" containerID="cri-o://3d0d9132064b1cd62436fc80c5bc75111a1ff534730e9f63b17321577836678d" gracePeriod=30 Nov 03 21:49:22.616702 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.616327 5041 generic.go:356] "Generic (PLEG): container finished" podID="9507b5d4-33e4-470f-9385-59164cb2e72a" containerID="d4a7661eb514e5931ceb633572b4ab35f215cd778792a6cc46fd057c387b92ce" exitCode=0 Nov 03 21:49:22.616702 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.616506 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" event={"ID":"9507b5d4-33e4-470f-9385-59164cb2e72a","Type":"ContainerDied","Data":"d4a7661eb514e5931ceb633572b4ab35f215cd778792a6cc46fd057c387b92ce"} Nov 03 21:49:22.616702 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.616577 5041 scope.go:122] "RemoveContainer" containerID="e6787f9d10bc60d58558aa3e2139d296b0b058d4dc1abe95cba826be5e3226f9" Nov 03 21:49:22.621600 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.621434 5041 generic.go:356] "Generic (PLEG): container finished" podID="277d0d55-608c-451a-a704-0a80454055bc" containerID="bda53a79e99c30d00c517ce9375f3a9f4644f5469301dd2a173d3f70df0c83df" exitCode=0 Nov 03 21:49:22.621600 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.621531 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" event={"ID":"277d0d55-608c-451a-a704-0a80454055bc","Type":"ContainerDied","Data":"bda53a79e99c30d00c517ce9375f3a9f4644f5469301dd2a173d3f70df0c83df"} Nov 03 21:49:22.633180 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.633100 5041 generic.go:356] "Generic (PLEG): container finished" podID="d82b63ac-65d4-4e30-8b51-d308cb3d335d" containerID="3d0d9132064b1cd62436fc80c5bc75111a1ff534730e9f63b17321577836678d" exitCode=0 Nov 03 21:49:22.640192 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.640165 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="259c779d-03a3-46bb-9091-ba214b339c72" path="/var/lib/kubelet/pods/259c779d-03a3-46bb-9091-ba214b339c72/volumes" Nov 03 21:49:22.656338 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.642886 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46" path="/var/lib/kubelet/pods/52c3dbd5-96d7-4b66-a3a3-22f0de7b3c46/volumes" Nov 03 21:49:22.656338 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.644076 5041 generic.go:356] "Generic (PLEG): container finished" podID="b68bf331-de7c-4e66-9371-72984471e65e" containerID="8377c76c2c5611506b0db9607b8ecb564ec0325f748ef853b32baeb6f123852b" exitCode=0 Nov 03 21:49:22.656338 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.644482 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d575fdb56-8br4h" event={"ID":"d82b63ac-65d4-4e30-8b51-d308cb3d335d","Type":"ContainerDied","Data":"3d0d9132064b1cd62436fc80c5bc75111a1ff534730e9f63b17321577836678d"} Nov 03 21:49:22.656338 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.644542 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" event={"ID":"b68bf331-de7c-4e66-9371-72984471e65e","Type":"ContainerDied","Data":"8377c76c2c5611506b0db9607b8ecb564ec0325f748ef853b32baeb6f123852b"} Nov 03 21:49:22.724093 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.720742 5041 scope.go:122] "RemoveContainer" containerID="e6f611effcbffbd5ced79fc870a015b984cc5d508dc3259d208460ee36d03206" Nov 03 21:49:22.947162 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.926483 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:49:23.011718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.973005 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-workload-certs\") pod \"b68bf331-de7c-4e66-9371-72984471e65e\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " Nov 03 21:49:23.011718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.973141 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68bf331-de7c-4e66-9371-72984471e65e-kube-api-access-6js8r\" (UniqueName: \"kubernetes.io/projected/b68bf331-de7c-4e66-9371-72984471e65e-kube-api-access-6js8r\") pod \"b68bf331-de7c-4e66-9371-72984471e65e\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " Nov 03 21:49:23.011718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.973187 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-workload-socket\") pod \"b68bf331-de7c-4e66-9371-72984471e65e\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " Nov 03 21:49:23.011718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.973236 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68bf331-de7c-4e66-9371-72984471e65e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68bf331-de7c-4e66-9371-72984471e65e-istiod-ca-cert\") pod \"b68bf331-de7c-4e66-9371-72984471e65e\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " Nov 03 21:49:23.011718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.973265 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-credential-socket\") pod \"b68bf331-de7c-4e66-9371-72984471e65e\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " Nov 03 21:49:23.011718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.973306 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68bf331-de7c-4e66-9371-72984471e65e-istio-token\" (UniqueName: \"kubernetes.io/projected/b68bf331-de7c-4e66-9371-72984471e65e-istio-token\") pod \"b68bf331-de7c-4e66-9371-72984471e65e\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " Nov 03 21:49:23.011718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.973339 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-istio-envoy\") pod \"b68bf331-de7c-4e66-9371-72984471e65e\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " Nov 03 21:49:23.011718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.973433 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b68bf331-de7c-4e66-9371-72984471e65e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68bf331-de7c-4e66-9371-72984471e65e-istio-podinfo\") pod \"b68bf331-de7c-4e66-9371-72984471e65e\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " Nov 03 21:49:23.011718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:22.973514 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-istio-data\") pod \"b68bf331-de7c-4e66-9371-72984471e65e\" (UID: \"b68bf331-de7c-4e66-9371-72984471e65e\") " Nov 03 21:49:23.014499 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.014139 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-workload-socket" pod "b68bf331-de7c-4e66-9371-72984471e65e" (UID: "b68bf331-de7c-4e66-9371-72984471e65e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:23.015030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.014762 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-credential-socket" pod "b68bf331-de7c-4e66-9371-72984471e65e" (UID: "b68bf331-de7c-4e66-9371-72984471e65e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:23.015503 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.015196 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-istio-data" pod "b68bf331-de7c-4e66-9371-72984471e65e" (UID: "b68bf331-de7c-4e66-9371-72984471e65e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:23.034815 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.034753 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-workload-certs" pod "b68bf331-de7c-4e66-9371-72984471e65e" (UID: "b68bf331-de7c-4e66-9371-72984471e65e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:23.077689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.065265 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68bf331-de7c-4e66-9371-72984471e65e-istiod-ca-cert" pod "b68bf331-de7c-4e66-9371-72984471e65e" (UID: "b68bf331-de7c-4e66-9371-72984471e65e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:49:23.088755 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.079045 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.088755 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.079084 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68bf331-de7c-4e66-9371-72984471e65e-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.088755 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.079100 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.088755 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.079116 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.088755 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.079132 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.116209 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.116149 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68bf331-de7c-4e66-9371-72984471e65e-kube-api-access-6js8r" pod "b68bf331-de7c-4e66-9371-72984471e65e" (UID: "b68bf331-de7c-4e66-9371-72984471e65e"). InnerVolumeSpecName "kube-api-access-6js8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:23.158184 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.157811 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b68bf331-de7c-4e66-9371-72984471e65e-istio-podinfo" pod "b68bf331-de7c-4e66-9371-72984471e65e" (UID: "b68bf331-de7c-4e66-9371-72984471e65e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:49:23.197685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.181027 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6js8r\" (UniqueName: \"kubernetes.io/projected/b68bf331-de7c-4e66-9371-72984471e65e-kube-api-access-6js8r\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.197685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.181065 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68bf331-de7c-4e66-9371-72984471e65e-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.302082 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.302047 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:49:23.389714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.363887 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:49:23.418734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.390280 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-token\" (UniqueName: \"kubernetes.io/projected/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-token\") pod \"9507b5d4-33e4-470f-9385-59164cb2e72a\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " Nov 03 21:49:23.418734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.390442 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-envoy\") pod \"9507b5d4-33e4-470f-9385-59164cb2e72a\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " Nov 03 21:49:23.418734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.390494 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-workload-socket\") pod \"9507b5d4-33e4-470f-9385-59164cb2e72a\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " Nov 03 21:49:23.418734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.390524 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9507b5d4-33e4-470f-9385-59164cb2e72a-kube-api-access-lcg6n\" (UniqueName: \"kubernetes.io/projected/9507b5d4-33e4-470f-9385-59164cb2e72a-kube-api-access-lcg6n\") pod \"9507b5d4-33e4-470f-9385-59164cb2e72a\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " Nov 03 21:49:23.418734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.390558 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9507b5d4-33e4-470f-9385-59164cb2e72a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9507b5d4-33e4-470f-9385-59164cb2e72a-istiod-ca-cert\") pod \"9507b5d4-33e4-470f-9385-59164cb2e72a\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " Nov 03 21:49:23.418734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.390626 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-data\") pod \"9507b5d4-33e4-470f-9385-59164cb2e72a\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " Nov 03 21:49:23.418734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.390706 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-podinfo\") pod \"9507b5d4-33e4-470f-9385-59164cb2e72a\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " Nov 03 21:49:23.418734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.390741 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-workload-certs\") pod \"9507b5d4-33e4-470f-9385-59164cb2e72a\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " Nov 03 21:49:23.418734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.414859 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-istio-envoy" pod "b68bf331-de7c-4e66-9371-72984471e65e" (UID: "b68bf331-de7c-4e66-9371-72984471e65e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:23.545685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.534874 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-credential-socket\") pod \"9507b5d4-33e4-470f-9385-59164cb2e72a\" (UID: \"9507b5d4-33e4-470f-9385-59164cb2e72a\") " Nov 03 21:49:23.545685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.535398 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68bf331-de7c-4e66-9371-72984471e65e-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.569693 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.566949 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-credential-socket" pod "9507b5d4-33e4-470f-9385-59164cb2e72a" (UID: "9507b5d4-33e4-470f-9385-59164cb2e72a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:23.576086 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.575897 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-workload-certs" pod "9507b5d4-33e4-470f-9385-59164cb2e72a" (UID: "9507b5d4-33e4-470f-9385-59164cb2e72a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:23.579684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.576295 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-workload-socket" pod "9507b5d4-33e4-470f-9385-59164cb2e72a" (UID: "9507b5d4-33e4-470f-9385-59164cb2e72a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:23.585686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.579887 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-data" pod "9507b5d4-33e4-470f-9385-59164cb2e72a" (UID: "9507b5d4-33e4-470f-9385-59164cb2e72a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:23.610687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.594769 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9507b5d4-33e4-470f-9385-59164cb2e72a-istiod-ca-cert" pod "9507b5d4-33e4-470f-9385-59164cb2e72a" (UID: "9507b5d4-33e4-470f-9385-59164cb2e72a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:49:23.650922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.625847 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-envoy" pod "9507b5d4-33e4-470f-9385-59164cb2e72a" (UID: "9507b5d4-33e4-470f-9385-59164cb2e72a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:23.650922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.625909 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-podinfo" pod "9507b5d4-33e4-470f-9385-59164cb2e72a" (UID: "9507b5d4-33e4-470f-9385-59164cb2e72a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:49:23.650922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.641106 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d82b63ac-65d4-4e30-8b51-d308cb3d335d-kube-api-access-2882x\" (UniqueName: \"kubernetes.io/projected/d82b63ac-65d4-4e30-8b51-d308cb3d335d-kube-api-access-2882x\") pod \"d82b63ac-65d4-4e30-8b51-d308cb3d335d\" (UID: \"d82b63ac-65d4-4e30-8b51-d308cb3d335d\") " Nov 03 21:49:23.650922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.641210 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d82b63ac-65d4-4e30-8b51-d308cb3d335d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/d82b63ac-65d4-4e30-8b51-d308cb3d335d-webhook-cert\") pod \"d82b63ac-65d4-4e30-8b51-d308cb3d335d\" (UID: \"d82b63ac-65d4-4e30-8b51-d308cb3d335d\") " Nov 03 21:49:23.650922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.641240 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d82b63ac-65d4-4e30-8b51-d308cb3d335d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82b63ac-65d4-4e30-8b51-d308cb3d335d-metrics-certs\") pod \"d82b63ac-65d4-4e30-8b51-d308cb3d335d\" (UID: \"d82b63ac-65d4-4e30-8b51-d308cb3d335d\") " Nov 03 21:49:23.650922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.641604 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.650922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.641625 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.650922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.641641 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9507b5d4-33e4-470f-9385-59164cb2e72a-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.650922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.641654 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.650922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.641685 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.650922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.641697 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.650922 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.641710 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9507b5d4-33e4-470f-9385-59164cb2e72a-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.808595 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.771124 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68bf331-de7c-4e66-9371-72984471e65e-istio-token" pod "b68bf331-de7c-4e66-9371-72984471e65e" (UID: "b68bf331-de7c-4e66-9371-72984471e65e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:23.808595 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.790414 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d82b63ac-65d4-4e30-8b51-d308cb3d335d-metrics-certs" pod "d82b63ac-65d4-4e30-8b51-d308cb3d335d" (UID: "d82b63ac-65d4-4e30-8b51-d308cb3d335d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:49:23.808595 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.796380 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-token" pod "9507b5d4-33e4-470f-9385-59164cb2e72a" (UID: "9507b5d4-33e4-470f-9385-59164cb2e72a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:23.808595 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.804797 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d82b63ac-65d4-4e30-8b51-d308cb3d335d-webhook-cert" pod "d82b63ac-65d4-4e30-8b51-d308cb3d335d" (UID: "d82b63ac-65d4-4e30-8b51-d308cb3d335d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:49:23.808595 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.808533 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9507b5d4-33e4-470f-9385-59164cb2e72a-kube-api-access-lcg6n" pod "9507b5d4-33e4-470f-9385-59164cb2e72a" (UID: "9507b5d4-33e4-470f-9385-59164cb2e72a"). InnerVolumeSpecName "kube-api-access-lcg6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:23.826475 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.810484 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82b63ac-65d4-4e30-8b51-d308cb3d335d-kube-api-access-2882x" pod "d82b63ac-65d4-4e30-8b51-d308cb3d335d" (UID: "d82b63ac-65d4-4e30-8b51-d308cb3d335d"). InnerVolumeSpecName "kube-api-access-2882x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:23.853774 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.828058 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" event={"ID":"9507b5d4-33e4-470f-9385-59164cb2e72a","Type":"ContainerDied","Data":"2a36549d77b928b62c7579a3bf23f9107b9fce1ff9c0712f5e968a12b5569d35"} Nov 03 21:49:23.853774 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.828132 5041 scope.go:122] "RemoveContainer" containerID="d4a7661eb514e5931ceb633572b4ab35f215cd778792a6cc46fd057c387b92ce" Nov 03 21:49:23.853774 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.828267 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq" Nov 03 21:49:23.854325 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.854154 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d575fdb56-8br4h" event={"ID":"d82b63ac-65d4-4e30-8b51-d308cb3d335d","Type":"ContainerDied","Data":"29904215fd7aa852143854a01d84967dd972a4cda5ad63b57373068817fdc8b3"} Nov 03 21:49:23.854325 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.854176 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d575fdb56-8br4h" Nov 03 21:49:23.865279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.860507 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" event={"ID":"b68bf331-de7c-4e66-9371-72984471e65e","Type":"ContainerDied","Data":"fe0e90c50c255fe633e1aae2b5fc636426cacd15a4a08a4ed8d82199cb8e105f"} Nov 03 21:49:23.865279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.860631 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc" Nov 03 21:49:23.865279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.863808 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:49:23.865279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.864467 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68bf331-de7c-4e66-9371-72984471e65e-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.865279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.864494 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2882x\" (UniqueName: \"kubernetes.io/projected/d82b63ac-65d4-4e30-8b51-d308cb3d335d-kube-api-access-2882x\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.865279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.864510 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcg6n\" (UniqueName: \"kubernetes.io/projected/9507b5d4-33e4-470f-9385-59164cb2e72a-kube-api-access-lcg6n\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.865279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.864535 5041 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d82b63ac-65d4-4e30-8b51-d308cb3d335d-webhook-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.865279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.864549 5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82b63ac-65d4-4e30-8b51-d308cb3d335d-metrics-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.865279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.864563 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9507b5d4-33e4-470f-9385-59164cb2e72a-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:23.965599 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:23.964883 5041 scope.go:122] "RemoveContainer" containerID="3d0d9132064b1cd62436fc80c5bc75111a1ff534730e9f63b17321577836678d" Nov 03 21:49:24.028059 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.027971 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq"] Nov 03 21:49:24.048708 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.048600 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-m8vtq"] Nov 03 21:49:24.106040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.084554 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-istio-envoy\") pod \"277d0d55-608c-451a-a704-0a80454055bc\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " Nov 03 21:49:24.106040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.084635 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-workload-socket\") pod \"277d0d55-608c-451a-a704-0a80454055bc\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " Nov 03 21:49:24.106040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.084713 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277d0d55-608c-451a-a704-0a80454055bc-kube-api-access-d7b9s\" (UniqueName: \"kubernetes.io/projected/277d0d55-608c-451a-a704-0a80454055bc-kube-api-access-d7b9s\") pod \"277d0d55-608c-451a-a704-0a80454055bc\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " Nov 03 21:49:24.106040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.084754 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277d0d55-608c-451a-a704-0a80454055bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277d0d55-608c-451a-a704-0a80454055bc-istiod-ca-cert\") pod \"277d0d55-608c-451a-a704-0a80454055bc\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " Nov 03 21:49:24.106040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.084807 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-istio-data\") pod \"277d0d55-608c-451a-a704-0a80454055bc\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " Nov 03 21:49:24.106040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.084923 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/277d0d55-608c-451a-a704-0a80454055bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277d0d55-608c-451a-a704-0a80454055bc-istio-podinfo\") pod \"277d0d55-608c-451a-a704-0a80454055bc\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " Nov 03 21:49:24.106040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.084988 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277d0d55-608c-451a-a704-0a80454055bc-istio-token\" (UniqueName: \"kubernetes.io/projected/277d0d55-608c-451a-a704-0a80454055bc-istio-token\") pod \"277d0d55-608c-451a-a704-0a80454055bc\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " Nov 03 21:49:24.106040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.085052 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-workload-certs\") pod \"277d0d55-608c-451a-a704-0a80454055bc\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " Nov 03 21:49:24.106040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.085154 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-credential-socket\") pod \"277d0d55-608c-451a-a704-0a80454055bc\" (UID: \"277d0d55-608c-451a-a704-0a80454055bc\") " Nov 03 21:49:24.106040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.093080 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277d0d55-608c-451a-a704-0a80454055bc-istiod-ca-cert" pod "277d0d55-608c-451a-a704-0a80454055bc" (UID: "277d0d55-608c-451a-a704-0a80454055bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:49:24.106040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.093625 5041 scope.go:122] "RemoveContainer" containerID="8377c76c2c5611506b0db9607b8ecb564ec0325f748ef853b32baeb6f123852b" Nov 03 21:49:24.106040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.094549 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-workload-socket" pod "277d0d55-608c-451a-a704-0a80454055bc" (UID: "277d0d55-608c-451a-a704-0a80454055bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:24.146687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.139107 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-istio-data" pod "277d0d55-608c-451a-a704-0a80454055bc" (UID: "277d0d55-608c-451a-a704-0a80454055bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:24.147055 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.146984 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-credential-socket" pod "277d0d55-608c-451a-a704-0a80454055bc" (UID: "277d0d55-608c-451a-a704-0a80454055bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:24.162779 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.162743 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x"] Nov 03 21:49:24.165283 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.165250 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-workload-certs" pod "277d0d55-608c-451a-a704-0a80454055bc" (UID: "277d0d55-608c-451a-a704-0a80454055bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:24.165793 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.165752 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-wc8rh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:49:24.166008 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.165962 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" podUID="44bc42a8-430e-44ec-b717-d1a2436968f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:49:24.197701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.196607 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:24.197701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.196650 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:24.197701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.196682 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:24.197701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.196732 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277d0d55-608c-451a-a704-0a80454055bc-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:24.197701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.196748 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:24.198200 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.198173 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-8b9jr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:49:24.198364 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.198338 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" podUID="69cc2bab-a8e9-4a4a-a114-88663140640d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:49:24.236708 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.236673 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad385913-6fn9x"] Nov 03 21:49:24.245794 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.239139 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-istio-envoy" pod "277d0d55-608c-451a-a704-0a80454055bc" (UID: "277d0d55-608c-451a-a704-0a80454055bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:49:24.261995 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.261941 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277d0d55-608c-451a-a704-0a80454055bc-istio-token" pod "277d0d55-608c-451a-a704-0a80454055bc" (UID: "277d0d55-608c-451a-a704-0a80454055bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:24.267818 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.267767 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/277d0d55-608c-451a-a704-0a80454055bc-istio-podinfo" pod "277d0d55-608c-451a-a704-0a80454055bc" (UID: "277d0d55-608c-451a-a704-0a80454055bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:49:24.331862 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.327438 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277d0d55-608c-451a-a704-0a80454055bc-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:24.331862 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.327469 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277d0d55-608c-451a-a704-0a80454055bc-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:24.331862 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.327486 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277d0d55-608c-451a-a704-0a80454055bc-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:24.331862 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.330529 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d575fdb56-8br4h"] Nov 03 21:49:24.378742 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.374710 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d575fdb56-8br4h"] Nov 03 21:49:24.400686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.398818 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277d0d55-608c-451a-a704-0a80454055bc-kube-api-access-d7b9s" pod "277d0d55-608c-451a-a704-0a80454055bc" (UID: "277d0d55-608c-451a-a704-0a80454055bc"). InnerVolumeSpecName "kube-api-access-d7b9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:24.431911 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.431380 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc"] Nov 03 21:49:24.432256 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.432105 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7b9s\" (UniqueName: \"kubernetes.io/projected/277d0d55-608c-451a-a704-0a80454055bc-kube-api-access-d7b9s\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:24.479058 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.479006 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-94qjc"] Nov 03 21:49:24.648194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.647755 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6kgrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:49:24.648194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.647845 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" podUID="bd662fa8-196b-4cad-b414-1bc2dbd68aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:49:24.698044 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.698005 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9507b5d4-33e4-470f-9385-59164cb2e72a" path="/var/lib/kubelet/pods/9507b5d4-33e4-470f-9385-59164cb2e72a/volumes" Nov 03 21:49:24.704605 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.704569 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68bf331-de7c-4e66-9371-72984471e65e" path="/var/lib/kubelet/pods/b68bf331-de7c-4e66-9371-72984471e65e/volumes" Nov 03 21:49:24.705467 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.705441 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30344d7-76ca-4674-a5d3-cf594b0ebdca" path="/var/lib/kubelet/pods/d30344d7-76ca-4674-a5d3-cf594b0ebdca/volumes" Nov 03 21:49:24.719734 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.719699 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d82b63ac-65d4-4e30-8b51-d308cb3d335d" path="/var/lib/kubelet/pods/d82b63ac-65d4-4e30-8b51-d308cb3d335d/volumes" Nov 03 21:49:24.957830 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.956548 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" event={"ID":"277d0d55-608c-451a-a704-0a80454055bc","Type":"ContainerDied","Data":"d675cc39093138ef1bdcf32bcc79e7113be7b474a44b48a700e48370a1434143"} Nov 03 21:49:24.957830 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.956625 5041 scope.go:122] "RemoveContainer" containerID="bda53a79e99c30d00c517ce9375f3a9f4644f5469301dd2a173d3f70df0c83df" Nov 03 21:49:24.957830 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:24.956829 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv" Nov 03 21:49:25.044706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.035380 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv"] Nov 03 21:49:25.097697 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.090039 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-517149-openshift-default-5cfb5c6655-cdjqv"] Nov 03 21:49:25.116110 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.116040 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68"] Nov 03 21:49:25.117220 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.117189 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47c651af-j8f68"] Nov 03 21:49:25.526689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.526629 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d575fdb56-mtkvs"] Nov 03 21:49:25.527229 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527208 5041 cpu_manager.go:414] "containerMap: removing container" podUID="9507b5d4-33e4-470f-9385-59164cb2e72a" containerName="istio-proxy" Nov 03 21:49:25.527328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527230 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9507b5d4-33e4-470f-9385-59164cb2e72a" containerName="istio-proxy" Nov 03 21:49:25.527328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527245 5041 cpu_manager.go:414] "containerMap: removing container" podUID="5150c7e7-763a-4b01-b493-d4046712ff6d" containerName="revoke-keys" Nov 03 21:49:25.527328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527252 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5150c7e7-763a-4b01-b493-d4046712ff6d" containerName="revoke-keys" Nov 03 21:49:25.527328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527270 5041 cpu_manager.go:414] "containerMap: removing container" podUID="d82b63ac-65d4-4e30-8b51-d308cb3d335d" containerName="manager" Nov 03 21:49:25.527328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527278 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d82b63ac-65d4-4e30-8b51-d308cb3d335d" containerName="manager" Nov 03 21:49:25.527328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527294 5041 cpu_manager.go:414] "containerMap: removing container" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" Nov 03 21:49:25.527328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527301 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" Nov 03 21:49:25.527328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527311 5041 cpu_manager.go:414] "containerMap: removing container" podUID="277d0d55-608c-451a-a704-0a80454055bc" containerName="istio-proxy" Nov 03 21:49:25.527328 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527318 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277d0d55-608c-451a-a704-0a80454055bc" containerName="istio-proxy" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527332 5041 cpu_manager.go:414] "containerMap: removing container" podUID="9507b5d4-33e4-470f-9385-59164cb2e72a" containerName="istio-proxy" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527339 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9507b5d4-33e4-470f-9385-59164cb2e72a" containerName="istio-proxy" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527353 5041 cpu_manager.go:414] "containerMap: removing container" podUID="d82b63ac-65d4-4e30-8b51-d308cb3d335d" containerName="manager" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527360 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d82b63ac-65d4-4e30-8b51-d308cb3d335d" containerName="manager" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527370 5041 cpu_manager.go:414] "containerMap: removing container" podUID="b68bf331-de7c-4e66-9371-72984471e65e" containerName="istio-proxy" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527377 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68bf331-de7c-4e66-9371-72984471e65e" containerName="istio-proxy" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527387 5041 cpu_manager.go:414] "containerMap: removing container" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527393 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527443 5041 cpu_manager.go:414] "containerMap: removing container" podUID="20b45af0-efad-4f42-a998-784bf9f39be9" containerName="payload-pre-processing" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527451 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b45af0-efad-4f42-a998-784bf9f39be9" containerName="payload-pre-processing" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527588 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b45af0-efad-4f42-a998-784bf9f39be9" containerName="payload-pre-processing" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527601 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9507b5d4-33e4-470f-9385-59164cb2e72a" containerName="istio-proxy" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527612 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5150c7e7-763a-4b01-b493-d4046712ff6d" containerName="revoke-keys" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527621 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d82b63ac-65d4-4e30-8b51-d308cb3d335d" containerName="manager" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527631 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="176dd3a2-8222-4882-97d7-27e51c5cd526" containerName="main" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527640 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="277d0d55-608c-451a-a704-0a80454055bc" containerName="istio-proxy" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527650 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9507b5d4-33e4-470f-9385-59164cb2e72a" containerName="istio-proxy" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527673 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d82b63ac-65d4-4e30-8b51-d308cb3d335d" containerName="manager" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527683 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68bf331-de7c-4e66-9371-72984471e65e" containerName="istio-proxy" Nov 03 21:49:25.527834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.527690 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0208e182-3db8-43cc-a945-9cea424a6796" containerName="main" Nov 03 21:49:25.533331 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.533311 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d575fdb56-mtkvs" Nov 03 21:49:25.554985 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.554956 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d575fdb56-mtkvs"] Nov 03 21:49:25.561613 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.561584 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cknnf\"" Nov 03 21:49:25.561613 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.561587 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 21:49:25.562926 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.562905 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 21:49:25.563037 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.563017 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 21:49:25.605524 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.605485 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-webhook-cert\") pod \"maas-controller-6d575fdb56-mtkvs\" (UID: \"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e\") " pod="opendatahub/maas-controller-6d575fdb56-mtkvs" Nov 03 21:49:25.605735 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.605710 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-metrics-certs\") pod \"maas-controller-6d575fdb56-mtkvs\" (UID: \"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e\") " pod="opendatahub/maas-controller-6d575fdb56-mtkvs" Nov 03 21:49:25.605813 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.605752 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd7cn\" (UniqueName: \"kubernetes.io/projected/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-kube-api-access-rd7cn\") pod \"maas-controller-6d575fdb56-mtkvs\" (UID: \"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e\") " pod="opendatahub/maas-controller-6d575fdb56-mtkvs" Nov 03 21:49:25.707109 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.707064 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-metrics-certs\") pod \"maas-controller-6d575fdb56-mtkvs\" (UID: \"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e\") " pod="opendatahub/maas-controller-6d575fdb56-mtkvs" Nov 03 21:49:25.707307 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.707117 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd7cn\" (UniqueName: \"kubernetes.io/projected/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-kube-api-access-rd7cn\") pod \"maas-controller-6d575fdb56-mtkvs\" (UID: \"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e\") " pod="opendatahub/maas-controller-6d575fdb56-mtkvs" Nov 03 21:49:25.707307 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.707216 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-webhook-cert\") pod \"maas-controller-6d575fdb56-mtkvs\" (UID: \"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e\") " pod="opendatahub/maas-controller-6d575fdb56-mtkvs" Nov 03 21:49:25.710850 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.710815 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-metrics-certs\") pod \"maas-controller-6d575fdb56-mtkvs\" (UID: \"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e\") " pod="opendatahub/maas-controller-6d575fdb56-mtkvs" Nov 03 21:49:25.711006 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.710972 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-webhook-cert\") pod \"maas-controller-6d575fdb56-mtkvs\" (UID: \"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e\") " pod="opendatahub/maas-controller-6d575fdb56-mtkvs" Nov 03 21:49:25.728269 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.728239 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd7cn\" (UniqueName: \"kubernetes.io/projected/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-kube-api-access-rd7cn\") pod \"maas-controller-6d575fdb56-mtkvs\" (UID: \"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e\") " pod="opendatahub/maas-controller-6d575fdb56-mtkvs" Nov 03 21:49:25.851554 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:25.851468 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d575fdb56-mtkvs" Nov 03 21:49:26.163701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:26.161007 5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:49:26.163701 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:26.162888 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d575fdb56-mtkvs"] Nov 03 21:49:26.623889 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:26.623857 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277d0d55-608c-451a-a704-0a80454055bc" path="/var/lib/kubelet/pods/277d0d55-608c-451a-a704-0a80454055bc/volumes" Nov 03 21:49:26.624466 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:26.624438 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb5f8ada-8543-4578-a8d9-a05d590b9982" path="/var/lib/kubelet/pods/cb5f8ada-8543-4578-a8d9-a05d590b9982/volumes" Nov 03 21:49:26.999507 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:26.999471 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d575fdb56-mtkvs" event={"ID":"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e","Type":"ContainerStarted","Data":"4981de5bb71c221ef894599d86550ed3025710134f6ff776d150baf227788dbb"} Nov 03 21:49:26.999754 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:26.999529 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d575fdb56-mtkvs" event={"ID":"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e","Type":"ContainerStarted","Data":"3644c21c57c477821b537998641936bfecd96457121551f05a2a09a7c8149ab3"} Nov 03 21:49:26.999754 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:26.999559 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d575fdb56-mtkvs" Nov 03 21:49:27.034438 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:27.034402 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl"] Nov 03 21:49:27.038377 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:27.038350 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4168ce3-2r5gl"] Nov 03 21:49:27.045481 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:27.045433 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6d575fdb56-mtkvs" podStartSLOduration=1.601253214 podStartE2EDuration="2.04541985s" totalImagesPullingTime="444.166636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:27.043920382 +0000 UTC m=+1182.551148662" watchObservedRunningTime="2026-11-03 21:49:27.04541985 +0000 UTC m=+1182.552648130" Nov 03 21:49:28.027340 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:28.027299 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst"] Nov 03 21:49:28.040606 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:28.040575 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afffa16-69bst"] Nov 03 21:49:28.620805 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:28.620773 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479af13b-6f89-4a24-9fd5-619888b129cc" path="/var/lib/kubelet/pods/479af13b-6f89-4a24-9fd5-619888b129cc/volumes" Nov 03 21:49:28.621226 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:28.621212 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e4dad7-bdde-4909-af0d-4f8eb8f145e5" path="/var/lib/kubelet/pods/58e4dad7-bdde-4909-af0d-4f8eb8f145e5/volumes" Nov 03 21:49:30.033043 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:30.033006 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6"] Nov 03 21:49:30.046562 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:30.046532 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c087459-ds7d6"] Nov 03 21:49:30.621700 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:30.621646 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a8d742e-582e-461c-93ce-26b511bfed39" path="/var/lib/kubelet/pods/0a8d742e-582e-461c-93ce-26b511bfed39/volumes" Nov 03 21:49:35.027869 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.027835 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr"] Nov 03 21:49:35.041125 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.041097 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e8f73e-d77qr"] Nov 03 21:49:35.117055 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.117018 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-s4cck-xtkdh"] Nov 03 21:49:35.121135 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.121119 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-s4cck-xtkdh" Nov 03 21:49:35.137609 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.137581 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lj4ds\"" Nov 03 21:49:35.142329 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.142299 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-s4cck-xtkdh"] Nov 03 21:49:35.208289 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.208257 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5b1060ff-0459-4df0-9b12-cb816a30d79b-archives-path\") pod \"periodic-gathering-s4cck-xtkdh\" (UID: \"5b1060ff-0459-4df0-9b12-cb816a30d79b\") " pod="openshift-insights/periodic-gathering-s4cck-xtkdh" Nov 03 21:49:35.208471 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.208328 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7t58\" (UniqueName: \"kubernetes.io/projected/5b1060ff-0459-4df0-9b12-cb816a30d79b-kube-api-access-c7t58\") pod \"periodic-gathering-s4cck-xtkdh\" (UID: \"5b1060ff-0459-4df0-9b12-cb816a30d79b\") " pod="openshift-insights/periodic-gathering-s4cck-xtkdh" Nov 03 21:49:35.208471 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.208394 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1060ff-0459-4df0-9b12-cb816a30d79b-service-ca-bundle\") pod \"periodic-gathering-s4cck-xtkdh\" (UID: \"5b1060ff-0459-4df0-9b12-cb816a30d79b\") " pod="openshift-insights/periodic-gathering-s4cck-xtkdh" Nov 03 21:49:35.208471 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.208424 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1060ff-0459-4df0-9b12-cb816a30d79b-trusted-ca-bundle\") pod \"periodic-gathering-s4cck-xtkdh\" (UID: \"5b1060ff-0459-4df0-9b12-cb816a30d79b\") " pod="openshift-insights/periodic-gathering-s4cck-xtkdh" Nov 03 21:49:35.309419 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.309341 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7t58\" (UniqueName: \"kubernetes.io/projected/5b1060ff-0459-4df0-9b12-cb816a30d79b-kube-api-access-c7t58\") pod \"periodic-gathering-s4cck-xtkdh\" (UID: \"5b1060ff-0459-4df0-9b12-cb816a30d79b\") " pod="openshift-insights/periodic-gathering-s4cck-xtkdh" Nov 03 21:49:35.309419 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.309378 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1060ff-0459-4df0-9b12-cb816a30d79b-service-ca-bundle\") pod \"periodic-gathering-s4cck-xtkdh\" (UID: \"5b1060ff-0459-4df0-9b12-cb816a30d79b\") " pod="openshift-insights/periodic-gathering-s4cck-xtkdh" Nov 03 21:49:35.309419 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.309404 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1060ff-0459-4df0-9b12-cb816a30d79b-trusted-ca-bundle\") pod \"periodic-gathering-s4cck-xtkdh\" (UID: \"5b1060ff-0459-4df0-9b12-cb816a30d79b\") " pod="openshift-insights/periodic-gathering-s4cck-xtkdh" Nov 03 21:49:35.309633 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.309482 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5b1060ff-0459-4df0-9b12-cb816a30d79b-archives-path\") pod \"periodic-gathering-s4cck-xtkdh\" (UID: \"5b1060ff-0459-4df0-9b12-cb816a30d79b\") " pod="openshift-insights/periodic-gathering-s4cck-xtkdh" Nov 03 21:49:35.309931 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.309913 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5b1060ff-0459-4df0-9b12-cb816a30d79b-archives-path\") pod \"periodic-gathering-s4cck-xtkdh\" (UID: \"5b1060ff-0459-4df0-9b12-cb816a30d79b\") " pod="openshift-insights/periodic-gathering-s4cck-xtkdh" Nov 03 21:49:35.310285 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.310259 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1060ff-0459-4df0-9b12-cb816a30d79b-service-ca-bundle\") pod \"periodic-gathering-s4cck-xtkdh\" (UID: \"5b1060ff-0459-4df0-9b12-cb816a30d79b\") " pod="openshift-insights/periodic-gathering-s4cck-xtkdh" Nov 03 21:49:35.310322 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.310309 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1060ff-0459-4df0-9b12-cb816a30d79b-trusted-ca-bundle\") pod \"periodic-gathering-s4cck-xtkdh\" (UID: \"5b1060ff-0459-4df0-9b12-cb816a30d79b\") " pod="openshift-insights/periodic-gathering-s4cck-xtkdh" Nov 03 21:49:35.330148 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.330123 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7t58\" (UniqueName: \"kubernetes.io/projected/5b1060ff-0459-4df0-9b12-cb816a30d79b-kube-api-access-c7t58\") pod \"periodic-gathering-s4cck-xtkdh\" (UID: \"5b1060ff-0459-4df0-9b12-cb816a30d79b\") " pod="openshift-insights/periodic-gathering-s4cck-xtkdh" Nov 03 21:49:35.439240 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.439206 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-s4cck-xtkdh" Nov 03 21:49:35.735086 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:49:35.735056 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1060ff_0459_4df0_9b12_cb816a30d79b.slice/crio-66e9b412329881719f5bb04fd760e70d98a327678e9d69ef26d216474c2b0521 WatchSource:0}: Error finding container 66e9b412329881719f5bb04fd760e70d98a327678e9d69ef26d216474c2b0521: Status 404 returned error can't find the container with id 66e9b412329881719f5bb04fd760e70d98a327678e9d69ef26d216474c2b0521 Nov 03 21:49:35.736871 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:35.736846 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-s4cck-xtkdh"] Nov 03 21:49:36.073699 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:36.073606 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-s4cck-xtkdh" event={"ID":"5b1060ff-0459-4df0-9b12-cb816a30d79b","Type":"ContainerStarted","Data":"1d515c83b2afaa8e63de977ead679f7991e8b16e9134f99556adf842e3266227"} Nov 03 21:49:36.073699 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:36.073656 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-s4cck-xtkdh" event={"ID":"5b1060ff-0459-4df0-9b12-cb816a30d79b","Type":"ContainerStarted","Data":"66e9b412329881719f5bb04fd760e70d98a327678e9d69ef26d216474c2b0521"} Nov 03 21:49:36.115763 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:36.115705 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-s4cck-xtkdh" podStartSLOduration=1.115687318 podStartE2EDuration="1.115687318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:36.113543759 +0000 UTC m=+1191.620772058" watchObservedRunningTime="2026-11-03 21:49:36.115687318 +0000 UTC m=+1191.622915599" Nov 03 21:49:36.621537 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:36.621501 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed16bde-a47b-48fd-a876-8836ef1d0ae7" path="/var/lib/kubelet/pods/7ed16bde-a47b-48fd-a876-8836ef1d0ae7/volumes" Nov 03 21:49:37.028891 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:37.028861 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw"] Nov 03 21:49:37.042592 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:37.042555 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv"] Nov 03 21:49:37.046373 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:37.046353 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04b00af2-rcdkw"] Nov 03 21:49:37.049517 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:37.049495 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b84cc7-lqdzv"] Nov 03 21:49:38.011179 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:38.011151 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d575fdb56-mtkvs" Nov 03 21:49:38.628790 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:38.628748 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a" path="/var/lib/kubelet/pods/25b76ee6-e687-4fe7-9ad0-67a6bbea2c7a/volumes" Nov 03 21:49:38.629279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:38.629256 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c23b00-eba0-44d0-b0ed-5c0300aca453" path="/var/lib/kubelet/pods/c7c23b00-eba0-44d0-b0ed-5c0300aca453/volumes" Nov 03 21:49:39.028915 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:39.028882 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn"] Nov 03 21:49:39.042236 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:39.042205 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6a7e516-8xgfn"] Nov 03 21:49:40.624577 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:40.624538 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa15f20-4f4b-4877-b1c9-84014f0dc01b" path="/var/lib/kubelet/pods/6fa15f20-4f4b-4877-b1c9-84014f0dc01b/volumes" Nov 03 21:49:40.777792 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:49:40.777758 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:49:41.027301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:41.027268 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2"] Nov 03 21:49:41.040679 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:41.040635 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4b2ad8-crqp2"] Nov 03 21:49:42.620825 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:42.620793 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa31e39-9530-4a6b-acaf-7e9eaf014929" path="/var/lib/kubelet/pods/3fa31e39-9530-4a6b-acaf-7e9eaf014929/volumes" Nov 03 21:49:43.298052 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:43.298021 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8"] Nov 03 21:49:43.303566 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:43.303543 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8" Nov 03 21:49:43.322695 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:43.322643 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8"] Nov 03 21:49:43.329711 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:43.329681 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:49:43.329711 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:43.329699 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:49:43.389232 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:43.389199 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn2t8\" (UniqueName: \"kubernetes.io/projected/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-kube-api-access-zn2t8\") pod \"maas-api-revoke-sub-85237e45-8lkg8\" (UID: \"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8" Nov 03 21:49:43.389400 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:43.389246 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8lkg8\" (UID: \"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8" Nov 03 21:49:43.389400 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:43.389270 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8lkg8\" (UID: \"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8" Nov 03 21:49:43.490756 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:43.490723 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn2t8\" (UniqueName: \"kubernetes.io/projected/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-kube-api-access-zn2t8\") pod \"maas-api-revoke-sub-85237e45-8lkg8\" (UID: \"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8" Nov 03 21:49:43.490959 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:43.490771 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8lkg8\" (UID: \"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8" Nov 03 21:49:43.490959 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:43.490869 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8lkg8\" (UID: \"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8" Nov 03 21:49:43.491557 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:43.491535 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8lkg8\" (UID: \"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8" Nov 03 21:49:43.493155 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:43.493135 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8lkg8\" (UID: \"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8" Nov 03 21:49:43.512443 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:43.512403 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn2t8\" (UniqueName: \"kubernetes.io/projected/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-kube-api-access-zn2t8\") pod \"maas-api-revoke-sub-85237e45-8lkg8\" (UID: \"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8" Nov 03 21:49:43.628386 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:43.628290 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8" Nov 03 21:49:43.930459 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:49:43.930431 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0ce3c5_169f_46c8_a3b7_6b2f61ddb68b.slice/crio-88d5f574ed5299b119adb71ed03ff78220fbca805ac1b143c66f26cf105afebe WatchSource:0}: Error finding container 88d5f574ed5299b119adb71ed03ff78220fbca805ac1b143c66f26cf105afebe: Status 404 returned error can't find the container with id 88d5f574ed5299b119adb71ed03ff78220fbca805ac1b143c66f26cf105afebe Nov 03 21:49:43.943702 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:43.933076 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8"] Nov 03 21:49:44.149526 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:44.149493 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8" event={"ID":"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b","Type":"ContainerStarted","Data":"88d5f574ed5299b119adb71ed03ff78220fbca805ac1b143c66f26cf105afebe"} Nov 03 21:49:45.161624 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:45.161550 5041 generic.go:356] "Generic (PLEG): container finished" podID="9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b" containerID="576e8eea534bb1793735305c1a9c3e1b2427193fa90cbb22dab8f38bc2878920" exitCode=0 Nov 03 21:49:45.161624 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:45.161586 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8" event={"ID":"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b","Type":"ContainerDied","Data":"576e8eea534bb1793735305c1a9c3e1b2427193fa90cbb22dab8f38bc2878920"} Nov 03 21:49:45.647084 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:45.647055 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 21:49:45.648857 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:45.648831 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 21:49:46.331687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:46.331621 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8" Nov 03 21:49:46.422919 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:46.422824 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-maas-api-xks-ca\") pod \"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b\" (UID: \"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b\") " Nov 03 21:49:46.423069 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:46.423047 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-maas-api-openshift-ca\") pod \"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b\" (UID: \"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b\") " Nov 03 21:49:46.423120 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:46.423101 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-kube-api-access-zn2t8\" (UniqueName: \"kubernetes.io/projected/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-kube-api-access-zn2t8\") pod \"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b\" (UID: \"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b\") " Nov 03 21:49:46.423389 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:46.423359 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-maas-api-openshift-ca" pod "9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b" (UID: "9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:49:46.423522 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:46.423453 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:46.424884 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:46.424852 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-maas-api-xks-ca" pod "9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b" (UID: "9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:49:46.425055 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:46.425035 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-kube-api-access-zn2t8" pod "9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b" (UID: "9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b"). InnerVolumeSpecName "kube-api-access-zn2t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:46.524594 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:46.524562 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn2t8\" (UniqueName: \"kubernetes.io/projected/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-kube-api-access-zn2t8\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:46.524594 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:46.524591 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:47.180378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:47.180350 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8" Nov 03 21:49:47.180546 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:47.180349 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8" event={"ID":"9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b","Type":"ContainerDied","Data":"88d5f574ed5299b119adb71ed03ff78220fbca805ac1b143c66f26cf105afebe"} Nov 03 21:49:47.180546 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:47.180467 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88d5f574ed5299b119adb71ed03ff78220fbca805ac1b143c66f26cf105afebe" Nov 03 21:49:55.217446 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:55.217408 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d575fdb56-mtkvs"] Nov 03 21:49:55.217920 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:55.217620 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6d575fdb56-mtkvs" podUID="a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e" containerName="manager" containerID="cri-o://4981de5bb71c221ef894599d86550ed3025710134f6ff776d150baf227788dbb" gracePeriod=30 Nov 03 21:49:55.482283 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:55.482224 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d575fdb56-mtkvs" Nov 03 21:49:55.613048 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:55.613015 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-webhook-cert\") pod \"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e\" (UID: \"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e\") " Nov 03 21:49:55.613228 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:55.613075 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-metrics-certs\") pod \"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e\" (UID: \"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e\") " Nov 03 21:49:55.613228 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:55.613126 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-kube-api-access-rd7cn\" (UniqueName: \"kubernetes.io/projected/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-kube-api-access-rd7cn\") pod \"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e\" (UID: \"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e\") " Nov 03 21:49:55.615151 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:55.615114 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-kube-api-access-rd7cn" pod "a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e" (UID: "a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e"). InnerVolumeSpecName "kube-api-access-rd7cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:49:55.615274 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:55.615168 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-metrics-certs" pod "a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e" (UID: "a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:49:55.615274 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:55.615217 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-webhook-cert" pod "a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e" (UID: "a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:49:55.715113 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:55.715083 5041 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-webhook-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:55.715113 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:55.715106 5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-metrics-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:55.715113 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:55.715116 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd7cn\" (UniqueName: \"kubernetes.io/projected/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e-kube-api-access-rd7cn\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:49:56.252815 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:56.252782 5041 generic.go:356] "Generic (PLEG): container finished" podID="a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e" containerID="4981de5bb71c221ef894599d86550ed3025710134f6ff776d150baf227788dbb" exitCode=0 Nov 03 21:49:56.253284 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:56.252827 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d575fdb56-mtkvs" Nov 03 21:49:56.253284 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:56.252872 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d575fdb56-mtkvs" event={"ID":"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e","Type":"ContainerDied","Data":"4981de5bb71c221ef894599d86550ed3025710134f6ff776d150baf227788dbb"} Nov 03 21:49:56.253284 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:56.252938 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d575fdb56-mtkvs" event={"ID":"a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e","Type":"ContainerDied","Data":"3644c21c57c477821b537998641936bfecd96457121551f05a2a09a7c8149ab3"} Nov 03 21:49:56.253284 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:56.252973 5041 scope.go:122] "RemoveContainer" containerID="4981de5bb71c221ef894599d86550ed3025710134f6ff776d150baf227788dbb" Nov 03 21:49:56.263979 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:56.263961 5041 scope.go:122] "RemoveContainer" containerID="4981de5bb71c221ef894599d86550ed3025710134f6ff776d150baf227788dbb" Nov 03 21:49:56.264231 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:49:56.264211 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4981de5bb71c221ef894599d86550ed3025710134f6ff776d150baf227788dbb\": container with ID starting with 4981de5bb71c221ef894599d86550ed3025710134f6ff776d150baf227788dbb not found: ID does not exist" containerID="4981de5bb71c221ef894599d86550ed3025710134f6ff776d150baf227788dbb" Nov 03 21:49:56.264284 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:56.264240 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4981de5bb71c221ef894599d86550ed3025710134f6ff776d150baf227788dbb"} err="failed to get container status \"4981de5bb71c221ef894599d86550ed3025710134f6ff776d150baf227788dbb\": rpc error: code = NotFound desc = could not find container \"4981de5bb71c221ef894599d86550ed3025710134f6ff776d150baf227788dbb\": container with ID starting with 4981de5bb71c221ef894599d86550ed3025710134f6ff776d150baf227788dbb not found: ID does not exist" Nov 03 21:49:56.314089 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:56.310158 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d575fdb56-mtkvs"] Nov 03 21:49:56.323566 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:56.323543 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d575fdb56-mtkvs"] Nov 03 21:49:56.620841 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:56.620768 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e" path="/var/lib/kubelet/pods/a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e/volumes" Nov 03 21:49:57.531843 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.531808 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d575fdb56-jwwm2"] Nov 03 21:49:57.532314 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.532302 5041 cpu_manager.go:414] "containerMap: removing container" podUID="a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e" containerName="manager" Nov 03 21:49:57.532360 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.532315 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e" containerName="manager" Nov 03 21:49:57.532360 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.532334 5041 cpu_manager.go:414] "containerMap: removing container" podUID="9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b" containerName="revoke-keys" Nov 03 21:49:57.532360 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.532339 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b" containerName="revoke-keys" Nov 03 21:49:57.532447 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.532427 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5bbf4bb-dd3e-4584-ad76-de3a0f7d067e" containerName="manager" Nov 03 21:49:57.532447 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.532438 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b" containerName="revoke-keys" Nov 03 21:49:57.537147 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.537130 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d575fdb56-jwwm2" Nov 03 21:49:57.567651 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.567615 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cknnf\"" Nov 03 21:49:57.567799 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.567684 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 21:49:57.567799 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.567714 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 21:49:57.567799 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.567782 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 21:49:57.571730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.571706 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d575fdb56-jwwm2"] Nov 03 21:49:57.631891 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.631865 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89259ba9-9d01-4e60-a875-9b02fb8bd98f-webhook-cert\") pod \"maas-controller-6d575fdb56-jwwm2\" (UID: \"89259ba9-9d01-4e60-a875-9b02fb8bd98f\") " pod="opendatahub/maas-controller-6d575fdb56-jwwm2" Nov 03 21:49:57.632002 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.631907 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlsk9\" (UniqueName: \"kubernetes.io/projected/89259ba9-9d01-4e60-a875-9b02fb8bd98f-kube-api-access-rlsk9\") pod \"maas-controller-6d575fdb56-jwwm2\" (UID: \"89259ba9-9d01-4e60-a875-9b02fb8bd98f\") " pod="opendatahub/maas-controller-6d575fdb56-jwwm2" Nov 03 21:49:57.632054 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.632004 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89259ba9-9d01-4e60-a875-9b02fb8bd98f-metrics-certs\") pod \"maas-controller-6d575fdb56-jwwm2\" (UID: \"89259ba9-9d01-4e60-a875-9b02fb8bd98f\") " pod="opendatahub/maas-controller-6d575fdb56-jwwm2" Nov 03 21:49:57.733567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.733536 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89259ba9-9d01-4e60-a875-9b02fb8bd98f-webhook-cert\") pod \"maas-controller-6d575fdb56-jwwm2\" (UID: \"89259ba9-9d01-4e60-a875-9b02fb8bd98f\") " pod="opendatahub/maas-controller-6d575fdb56-jwwm2" Nov 03 21:49:57.733755 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.733586 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlsk9\" (UniqueName: \"kubernetes.io/projected/89259ba9-9d01-4e60-a875-9b02fb8bd98f-kube-api-access-rlsk9\") pod \"maas-controller-6d575fdb56-jwwm2\" (UID: \"89259ba9-9d01-4e60-a875-9b02fb8bd98f\") " pod="opendatahub/maas-controller-6d575fdb56-jwwm2" Nov 03 21:49:57.733828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.733747 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89259ba9-9d01-4e60-a875-9b02fb8bd98f-metrics-certs\") pod \"maas-controller-6d575fdb56-jwwm2\" (UID: \"89259ba9-9d01-4e60-a875-9b02fb8bd98f\") " pod="opendatahub/maas-controller-6d575fdb56-jwwm2" Nov 03 21:49:57.736030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.736003 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89259ba9-9d01-4e60-a875-9b02fb8bd98f-webhook-cert\") pod \"maas-controller-6d575fdb56-jwwm2\" (UID: \"89259ba9-9d01-4e60-a875-9b02fb8bd98f\") " pod="opendatahub/maas-controller-6d575fdb56-jwwm2" Nov 03 21:49:57.736158 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.736139 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89259ba9-9d01-4e60-a875-9b02fb8bd98f-metrics-certs\") pod \"maas-controller-6d575fdb56-jwwm2\" (UID: \"89259ba9-9d01-4e60-a875-9b02fb8bd98f\") " pod="opendatahub/maas-controller-6d575fdb56-jwwm2" Nov 03 21:49:57.753717 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.753692 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlsk9\" (UniqueName: \"kubernetes.io/projected/89259ba9-9d01-4e60-a875-9b02fb8bd98f-kube-api-access-rlsk9\") pod \"maas-controller-6d575fdb56-jwwm2\" (UID: \"89259ba9-9d01-4e60-a875-9b02fb8bd98f\") " pod="opendatahub/maas-controller-6d575fdb56-jwwm2" Nov 03 21:49:57.850246 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:57.850174 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d575fdb56-jwwm2" Nov 03 21:49:58.145802 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:49:58.145774 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89259ba9_9d01_4e60_a875_9b02fb8bd98f.slice/crio-714413c6f9fef456350a39de573599da1a6f5f2cbb109b2b3af697fb6215ed8f WatchSource:0}: Error finding container 714413c6f9fef456350a39de573599da1a6f5f2cbb109b2b3af697fb6215ed8f: Status 404 returned error can't find the container with id 714413c6f9fef456350a39de573599da1a6f5f2cbb109b2b3af697fb6215ed8f Nov 03 21:49:58.145919 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:58.145864 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d575fdb56-jwwm2"] Nov 03 21:49:58.270219 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:58.270185 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d575fdb56-jwwm2" event={"ID":"89259ba9-9d01-4e60-a875-9b02fb8bd98f","Type":"ContainerStarted","Data":"714413c6f9fef456350a39de573599da1a6f5f2cbb109b2b3af697fb6215ed8f"} Nov 03 21:49:59.281134 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:59.281092 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d575fdb56-jwwm2" event={"ID":"89259ba9-9d01-4e60-a875-9b02fb8bd98f","Type":"ContainerStarted","Data":"10e169b9e6acdf359ade699c7cb72be8d203fac69b5d779f9e14302834b59365"} Nov 03 21:49:59.281619 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:59.281593 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d575fdb56-jwwm2" Nov 03 21:49:59.322571 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:49:59.322511 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6d575fdb56-jwwm2" podStartSLOduration=2.001343825 podStartE2EDuration="2.322493646s" totalImagesPullingTime="321.149821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:59.32176365 +0000 UTC m=+1214.828991931" watchObservedRunningTime="2026-11-03 21:49:59.322493646 +0000 UTC m=+1214.829721928" Nov 03 21:50:06.027815 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:06.027781 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9"] Nov 03 21:50:06.036203 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:06.036175 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21af73a2-nwll9"] Nov 03 21:50:06.628638 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:06.628601 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9256e44-c54f-4801-9cd7-bec18a050043" path="/var/lib/kubelet/pods/b9256e44-c54f-4801-9cd7-bec18a050043/volumes" Nov 03 21:50:10.294075 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:10.294039 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d575fdb56-jwwm2" Nov 03 21:50:10.777897 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:50:10.777861 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:50:20.028202 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:20.028159 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62"] Nov 03 21:50:20.041208 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:20.041175 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d838950-9fw62"] Nov 03 21:50:20.621528 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:20.621488 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733c68bd-9bf7-46ee-8b4f-b1aa3144fa19" path="/var/lib/kubelet/pods/733c68bd-9bf7-46ee-8b4f-b1aa3144fa19/volumes" Nov 03 21:50:40.776436 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:50:40.776397 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:50:41.022332 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.022301 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb"] Nov 03 21:50:41.026277 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.026255 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb" Nov 03 21:50:41.046567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.046526 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb"] Nov 03 21:50:41.056281 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.056254 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:50:41.056281 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.056254 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:50:41.139805 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.139767 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b98d0a76-ce06-45e6-9040-204b7b95a5bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-gsqkb\" (UID: \"b98d0a76-ce06-45e6-9040-204b7b95a5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb" Nov 03 21:50:41.139980 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.139815 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b98d0a76-ce06-45e6-9040-204b7b95a5bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-gsqkb\" (UID: \"b98d0a76-ce06-45e6-9040-204b7b95a5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb" Nov 03 21:50:41.140021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.139995 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjjch\" (UniqueName: \"kubernetes.io/projected/b98d0a76-ce06-45e6-9040-204b7b95a5bb-kube-api-access-xjjch\") pod \"maas-api-revoke-sub-c1e271a9-gsqkb\" (UID: \"b98d0a76-ce06-45e6-9040-204b7b95a5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb" Nov 03 21:50:41.240610 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.240573 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b98d0a76-ce06-45e6-9040-204b7b95a5bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-gsqkb\" (UID: \"b98d0a76-ce06-45e6-9040-204b7b95a5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb" Nov 03 21:50:41.240835 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.240618 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b98d0a76-ce06-45e6-9040-204b7b95a5bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-gsqkb\" (UID: \"b98d0a76-ce06-45e6-9040-204b7b95a5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb" Nov 03 21:50:41.240835 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.240809 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjjch\" (UniqueName: \"kubernetes.io/projected/b98d0a76-ce06-45e6-9040-204b7b95a5bb-kube-api-access-xjjch\") pod \"maas-api-revoke-sub-c1e271a9-gsqkb\" (UID: \"b98d0a76-ce06-45e6-9040-204b7b95a5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb" Nov 03 21:50:41.241404 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.241374 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b98d0a76-ce06-45e6-9040-204b7b95a5bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-gsqkb\" (UID: \"b98d0a76-ce06-45e6-9040-204b7b95a5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb" Nov 03 21:50:41.243247 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.243222 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b98d0a76-ce06-45e6-9040-204b7b95a5bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-gsqkb\" (UID: \"b98d0a76-ce06-45e6-9040-204b7b95a5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb" Nov 03 21:50:41.261130 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.261102 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjjch\" (UniqueName: \"kubernetes.io/projected/b98d0a76-ce06-45e6-9040-204b7b95a5bb-kube-api-access-xjjch\") pod \"maas-api-revoke-sub-c1e271a9-gsqkb\" (UID: \"b98d0a76-ce06-45e6-9040-204b7b95a5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb" Nov 03 21:50:41.340865 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.340774 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb" Nov 03 21:50:41.638397 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:50:41.638365 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98d0a76_ce06_45e6_9040_204b7b95a5bb.slice/crio-eb88da071186ca4250c62f94b9d6f54c5da8e67ec1c972dac1fac0890f22b74d WatchSource:0}: Error finding container eb88da071186ca4250c62f94b9d6f54c5da8e67ec1c972dac1fac0890f22b74d: Status 404 returned error can't find the container with id eb88da071186ca4250c62f94b9d6f54c5da8e67ec1c972dac1fac0890f22b74d Nov 03 21:50:41.638535 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.638471 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb"] Nov 03 21:50:41.730694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:41.730652 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb" event={"ID":"b98d0a76-ce06-45e6-9040-204b7b95a5bb","Type":"ContainerStarted","Data":"eb88da071186ca4250c62f94b9d6f54c5da8e67ec1c972dac1fac0890f22b74d"} Nov 03 21:50:42.740886 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:42.740849 5041 generic.go:356] "Generic (PLEG): container finished" podID="b98d0a76-ce06-45e6-9040-204b7b95a5bb" containerID="e51dc825d2eae148d3e39faa8637f787f23208d35d002298be17797ea732411e" exitCode=0 Nov 03 21:50:42.741308 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:42.740934 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb" event={"ID":"b98d0a76-ce06-45e6-9040-204b7b95a5bb","Type":"ContainerDied","Data":"e51dc825d2eae148d3e39faa8637f787f23208d35d002298be17797ea732411e"} Nov 03 21:50:43.895392 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:43.895363 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb" Nov 03 21:50:43.969718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:43.969679 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b98d0a76-ce06-45e6-9040-204b7b95a5bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b98d0a76-ce06-45e6-9040-204b7b95a5bb-maas-api-openshift-ca\") pod \"b98d0a76-ce06-45e6-9040-204b7b95a5bb\" (UID: \"b98d0a76-ce06-45e6-9040-204b7b95a5bb\") " Nov 03 21:50:43.969912 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:43.969750 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98d0a76-ce06-45e6-9040-204b7b95a5bb-kube-api-access-xjjch\" (UniqueName: \"kubernetes.io/projected/b98d0a76-ce06-45e6-9040-204b7b95a5bb-kube-api-access-xjjch\") pod \"b98d0a76-ce06-45e6-9040-204b7b95a5bb\" (UID: \"b98d0a76-ce06-45e6-9040-204b7b95a5bb\") " Nov 03 21:50:43.969912 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:43.969878 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98d0a76-ce06-45e6-9040-204b7b95a5bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b98d0a76-ce06-45e6-9040-204b7b95a5bb-maas-api-xks-ca\") pod \"b98d0a76-ce06-45e6-9040-204b7b95a5bb\" (UID: \"b98d0a76-ce06-45e6-9040-204b7b95a5bb\") " Nov 03 21:50:43.970152 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:43.970123 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98d0a76-ce06-45e6-9040-204b7b95a5bb-maas-api-openshift-ca" pod "b98d0a76-ce06-45e6-9040-204b7b95a5bb" (UID: "b98d0a76-ce06-45e6-9040-204b7b95a5bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:50:43.970333 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:43.970307 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b98d0a76-ce06-45e6-9040-204b7b95a5bb-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:50:43.972222 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:43.972193 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98d0a76-ce06-45e6-9040-204b7b95a5bb-kube-api-access-xjjch" pod "b98d0a76-ce06-45e6-9040-204b7b95a5bb" (UID: "b98d0a76-ce06-45e6-9040-204b7b95a5bb"). InnerVolumeSpecName "kube-api-access-xjjch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:50:43.973227 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:43.973196 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98d0a76-ce06-45e6-9040-204b7b95a5bb-maas-api-xks-ca" pod "b98d0a76-ce06-45e6-9040-204b7b95a5bb" (UID: "b98d0a76-ce06-45e6-9040-204b7b95a5bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:50:44.071921 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:44.071843 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjjch\" (UniqueName: \"kubernetes.io/projected/b98d0a76-ce06-45e6-9040-204b7b95a5bb-kube-api-access-xjjch\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:50:44.071921 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:44.071873 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b98d0a76-ce06-45e6-9040-204b7b95a5bb-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:50:44.757397 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:44.757362 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb" event={"ID":"b98d0a76-ce06-45e6-9040-204b7b95a5bb","Type":"ContainerDied","Data":"eb88da071186ca4250c62f94b9d6f54c5da8e67ec1c972dac1fac0890f22b74d"} Nov 03 21:50:44.757397 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:44.757397 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb" Nov 03 21:50:44.757608 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:44.757413 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb88da071186ca4250c62f94b9d6f54c5da8e67ec1c972dac1fac0890f22b74d" Nov 03 21:50:46.027906 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:46.027871 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt"] Nov 03 21:50:46.041506 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:46.041473 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b20ba06-ll5mt"] Nov 03 21:50:46.621511 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:46.621474 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35091bce-d209-4b19-a4f3-ce7707badfad" path="/var/lib/kubelet/pods/35091bce-d209-4b19-a4f3-ce7707badfad/volumes" Nov 03 21:50:47.748214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.748177 5041 scope.go:122] "RemoveContainer" containerID="a57f6fe153f0327b19ff7da3572c5b0a62723f9d675b99902e51a3786b6c45f6" Nov 03 21:50:47.761783 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.761758 5041 scope.go:122] "RemoveContainer" containerID="bfad6edb99945282362a118f4b219c12587322df2b077bb8ab82436574533ce5" Nov 03 21:50:47.772787 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.772767 5041 scope.go:122] "RemoveContainer" containerID="47218e364eb304fc56fe76e7eaebd86a491918dda9ff89764bf85bb2e4c17024" Nov 03 21:50:47.783244 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.783224 5041 scope.go:122] "RemoveContainer" containerID="4d69c270a51d1d6aac46c4a43aa9b16732203cda47fb9f8b1ade0a48c22a7c81" Nov 03 21:50:47.792916 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.792897 5041 scope.go:122] "RemoveContainer" containerID="5d1c21f6ba45de8a2edb87db00ebabea1862479701c06ff8f21ab8b1490d395b" Nov 03 21:50:47.803888 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.803868 5041 scope.go:122] "RemoveContainer" containerID="835082d3140ca2344d8107ac398707486793123fdba3dad55029855519e1adf9" Nov 03 21:50:47.813049 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.813026 5041 scope.go:122] "RemoveContainer" containerID="54a8fb4b1849aa90bb45f12ac447bc281352e6f5393958bfdabc658e893567c7" Nov 03 21:50:47.821991 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.821970 5041 scope.go:122] "RemoveContainer" containerID="7e3039c2f39622feb0796bd4d2fac0275ffe5307d7441bdbabeb4b9b88e24ab8" Nov 03 21:50:47.831017 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.830998 5041 scope.go:122] "RemoveContainer" containerID="be087dba4989585f7bcdc9e5dc308bfd954d34674ec67f0d0c2c1908528bd884" Nov 03 21:50:47.839789 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.839751 5041 scope.go:122] "RemoveContainer" containerID="d24de2e036e77e0d63e860318e93c9bc66b96def4a2fddd1c486ec6ebbd43f53" Nov 03 21:50:47.849536 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.849516 5041 scope.go:122] "RemoveContainer" containerID="7b9ba69db9a5332f377eed9e6b450712957548fd6052da1ee03d443fd029b84f" Nov 03 21:50:47.858723 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.858657 5041 scope.go:122] "RemoveContainer" containerID="78a7f0e04c5d1c8b59c50b80fe599515aae7e2f4d8947800eb956f3930318ad9" Nov 03 21:50:47.868182 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.868158 5041 scope.go:122] "RemoveContainer" containerID="167855d778ef2384cdf85aebe2a10a382d83d8c60b3cfc9f2f13a14ba135d1ed" Nov 03 21:50:47.877416 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.877392 5041 scope.go:122] "RemoveContainer" containerID="f2c59208cd16c4713aeb0dd988641ecb74d7f24c9c99eb768dbf6ebe61c00e1c" Nov 03 21:50:47.886421 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.886399 5041 scope.go:122] "RemoveContainer" containerID="340ec8c87542889b8c191685c4b98a8e4805bafffbe0588f26b1e19ab01c5c38" Nov 03 21:50:47.895457 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.895436 5041 scope.go:122] "RemoveContainer" containerID="9c7eec835f12b2a6327e3be182622c8c87ef06040ee9c56906ddbd953105cfcb" Nov 03 21:50:47.904469 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:50:47.904445 5041 scope.go:122] "RemoveContainer" containerID="1ff9fab001a1700e83f684e8e9c7b90eaf63e1319532aeb5f6bb412fe9a8f823" Nov 03 21:51:04.276965 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.276930 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv"] Nov 03 21:51:04.277446 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.277432 5041 cpu_manager.go:414] "containerMap: removing container" podUID="b98d0a76-ce06-45e6-9040-204b7b95a5bb" containerName="revoke-keys" Nov 03 21:51:04.277502 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.277448 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98d0a76-ce06-45e6-9040-204b7b95a5bb" containerName="revoke-keys" Nov 03 21:51:04.277554 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.277545 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98d0a76-ce06-45e6-9040-204b7b95a5bb" containerName="revoke-keys" Nov 03 21:51:04.280892 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.280876 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv" Nov 03 21:51:04.300404 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.300370 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv"] Nov 03 21:51:04.310161 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.310130 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:51:04.310161 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.310146 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:51:04.335573 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.335538 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14b358c-04e7-42bf-b914-f41d424cad39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-t8brv\" (UID: \"c14b358c-04e7-42bf-b914-f41d424cad39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv" Nov 03 21:51:04.335770 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.335603 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14b358c-04e7-42bf-b914-f41d424cad39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-t8brv\" (UID: \"c14b358c-04e7-42bf-b914-f41d424cad39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv" Nov 03 21:51:04.335817 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.335782 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cblw\" (UniqueName: \"kubernetes.io/projected/c14b358c-04e7-42bf-b914-f41d424cad39-kube-api-access-4cblw\") pod \"maas-api-revoke-sub-2e1f5b5f-t8brv\" (UID: \"c14b358c-04e7-42bf-b914-f41d424cad39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv" Nov 03 21:51:04.436730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.436655 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cblw\" (UniqueName: \"kubernetes.io/projected/c14b358c-04e7-42bf-b914-f41d424cad39-kube-api-access-4cblw\") pod \"maas-api-revoke-sub-2e1f5b5f-t8brv\" (UID: \"c14b358c-04e7-42bf-b914-f41d424cad39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv" Nov 03 21:51:04.436926 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.436775 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14b358c-04e7-42bf-b914-f41d424cad39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-t8brv\" (UID: \"c14b358c-04e7-42bf-b914-f41d424cad39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv" Nov 03 21:51:04.436926 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.436824 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14b358c-04e7-42bf-b914-f41d424cad39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-t8brv\" (UID: \"c14b358c-04e7-42bf-b914-f41d424cad39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv" Nov 03 21:51:04.437608 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.437578 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14b358c-04e7-42bf-b914-f41d424cad39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-t8brv\" (UID: \"c14b358c-04e7-42bf-b914-f41d424cad39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv" Nov 03 21:51:04.439197 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.439168 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14b358c-04e7-42bf-b914-f41d424cad39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-t8brv\" (UID: \"c14b358c-04e7-42bf-b914-f41d424cad39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv" Nov 03 21:51:04.456929 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.456900 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cblw\" (UniqueName: \"kubernetes.io/projected/c14b358c-04e7-42bf-b914-f41d424cad39-kube-api-access-4cblw\") pod \"maas-api-revoke-sub-2e1f5b5f-t8brv\" (UID: \"c14b358c-04e7-42bf-b914-f41d424cad39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv" Nov 03 21:51:04.595544 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.595462 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv" Nov 03 21:51:04.896276 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:51:04.896240 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14b358c_04e7_42bf_b914_f41d424cad39.slice/crio-081bffd7af6ea52280466a2d6e8cd6f55723217ccc9b22838f5914bdacd04879 WatchSource:0}: Error finding container 081bffd7af6ea52280466a2d6e8cd6f55723217ccc9b22838f5914bdacd04879: Status 404 returned error can't find the container with id 081bffd7af6ea52280466a2d6e8cd6f55723217ccc9b22838f5914bdacd04879 Nov 03 21:51:04.896432 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.896331 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv"] Nov 03 21:51:04.959211 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:04.959178 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv" event={"ID":"c14b358c-04e7-42bf-b914-f41d424cad39","Type":"ContainerStarted","Data":"081bffd7af6ea52280466a2d6e8cd6f55723217ccc9b22838f5914bdacd04879"} Nov 03 21:51:05.967272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:05.967237 5041 generic.go:356] "Generic (PLEG): container finished" podID="c14b358c-04e7-42bf-b914-f41d424cad39" containerID="bcace69a0b8a4be1020b8eadda93dd5157f15215930aad778197ee7f73a1b2d9" exitCode=0 Nov 03 21:51:05.967712 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:05.967290 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv" event={"ID":"c14b358c-04e7-42bf-b914-f41d424cad39","Type":"ContainerDied","Data":"bcace69a0b8a4be1020b8eadda93dd5157f15215930aad778197ee7f73a1b2d9"} Nov 03 21:51:07.124546 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:07.124514 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv" Nov 03 21:51:07.166602 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:07.166572 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c14b358c-04e7-42bf-b914-f41d424cad39-kube-api-access-4cblw\" (UniqueName: \"kubernetes.io/projected/c14b358c-04e7-42bf-b914-f41d424cad39-kube-api-access-4cblw\") pod \"c14b358c-04e7-42bf-b914-f41d424cad39\" (UID: \"c14b358c-04e7-42bf-b914-f41d424cad39\") " Nov 03 21:51:07.166827 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:07.166720 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c14b358c-04e7-42bf-b914-f41d424cad39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14b358c-04e7-42bf-b914-f41d424cad39-maas-api-xks-ca\") pod \"c14b358c-04e7-42bf-b914-f41d424cad39\" (UID: \"c14b358c-04e7-42bf-b914-f41d424cad39\") " Nov 03 21:51:07.166827 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:07.166766 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c14b358c-04e7-42bf-b914-f41d424cad39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14b358c-04e7-42bf-b914-f41d424cad39-maas-api-openshift-ca\") pod \"c14b358c-04e7-42bf-b914-f41d424cad39\" (UID: \"c14b358c-04e7-42bf-b914-f41d424cad39\") " Nov 03 21:51:07.167358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:07.167329 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14b358c-04e7-42bf-b914-f41d424cad39-maas-api-openshift-ca" pod "c14b358c-04e7-42bf-b914-f41d424cad39" (UID: "c14b358c-04e7-42bf-b914-f41d424cad39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:51:07.168854 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:07.168829 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14b358c-04e7-42bf-b914-f41d424cad39-kube-api-access-4cblw" pod "c14b358c-04e7-42bf-b914-f41d424cad39" (UID: "c14b358c-04e7-42bf-b914-f41d424cad39"). InnerVolumeSpecName "kube-api-access-4cblw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:51:07.168953 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:07.168924 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c14b358c-04e7-42bf-b914-f41d424cad39-maas-api-xks-ca" pod "c14b358c-04e7-42bf-b914-f41d424cad39" (UID: "c14b358c-04e7-42bf-b914-f41d424cad39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:51:07.268487 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:07.268409 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cblw\" (UniqueName: \"kubernetes.io/projected/c14b358c-04e7-42bf-b914-f41d424cad39-kube-api-access-4cblw\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:51:07.268487 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:07.268437 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14b358c-04e7-42bf-b914-f41d424cad39-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:51:07.268487 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:07.268448 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14b358c-04e7-42bf-b914-f41d424cad39-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:51:07.983036 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:07.983005 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv" Nov 03 21:51:07.983185 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:07.983037 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv" event={"ID":"c14b358c-04e7-42bf-b914-f41d424cad39","Type":"ContainerDied","Data":"081bffd7af6ea52280466a2d6e8cd6f55723217ccc9b22838f5914bdacd04879"} Nov 03 21:51:07.983185 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:07.983084 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="081bffd7af6ea52280466a2d6e8cd6f55723217ccc9b22838f5914bdacd04879" Nov 03 21:51:09.999982 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:09.999948 5041 generic.go:356] "Generic (PLEG): container finished" podID="5b1060ff-0459-4df0-9b12-cb816a30d79b" containerID="1d515c83b2afaa8e63de977ead679f7991e8b16e9134f99556adf842e3266227" exitCode=0 Nov 03 21:51:10.000322 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:10.000021 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-s4cck-xtkdh" event={"ID":"5b1060ff-0459-4df0-9b12-cb816a30d79b","Type":"ContainerDied","Data":"1d515c83b2afaa8e63de977ead679f7991e8b16e9134f99556adf842e3266227"} Nov 03 21:51:10.776138 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:51:10.776101 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:51:11.154138 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:11.154110 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-s4cck-xtkdh" Nov 03 21:51:11.210026 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:11.209994 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b1060ff-0459-4df0-9b12-cb816a30d79b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1060ff-0459-4df0-9b12-cb816a30d79b-service-ca-bundle\") pod \"5b1060ff-0459-4df0-9b12-cb816a30d79b\" (UID: \"5b1060ff-0459-4df0-9b12-cb816a30d79b\") " Nov 03 21:51:11.210207 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:11.210042 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b1060ff-0459-4df0-9b12-cb816a30d79b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1060ff-0459-4df0-9b12-cb816a30d79b-trusted-ca-bundle\") pod \"5b1060ff-0459-4df0-9b12-cb816a30d79b\" (UID: \"5b1060ff-0459-4df0-9b12-cb816a30d79b\") " Nov 03 21:51:11.210207 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:11.210107 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b1060ff-0459-4df0-9b12-cb816a30d79b-kube-api-access-c7t58\" (UniqueName: \"kubernetes.io/projected/5b1060ff-0459-4df0-9b12-cb816a30d79b-kube-api-access-c7t58\") pod \"5b1060ff-0459-4df0-9b12-cb816a30d79b\" (UID: \"5b1060ff-0459-4df0-9b12-cb816a30d79b\") " Nov 03 21:51:11.210207 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:11.210155 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b1060ff-0459-4df0-9b12-cb816a30d79b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5b1060ff-0459-4df0-9b12-cb816a30d79b-archives-path\") pod \"5b1060ff-0459-4df0-9b12-cb816a30d79b\" (UID: \"5b1060ff-0459-4df0-9b12-cb816a30d79b\") " Nov 03 21:51:11.210515 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:11.210488 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1060ff-0459-4df0-9b12-cb816a30d79b-trusted-ca-bundle" pod "5b1060ff-0459-4df0-9b12-cb816a30d79b" (UID: "5b1060ff-0459-4df0-9b12-cb816a30d79b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:51:11.210567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:11.210529 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1060ff-0459-4df0-9b12-cb816a30d79b-service-ca-bundle" pod "5b1060ff-0459-4df0-9b12-cb816a30d79b" (UID: "5b1060ff-0459-4df0-9b12-cb816a30d79b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:51:11.210607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:11.210573 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1060ff-0459-4df0-9b12-cb816a30d79b-archives-path" pod "5b1060ff-0459-4df0-9b12-cb816a30d79b" (UID: "5b1060ff-0459-4df0-9b12-cb816a30d79b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:51:11.212647 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:11.212625 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1060ff-0459-4df0-9b12-cb816a30d79b-kube-api-access-c7t58" pod "5b1060ff-0459-4df0-9b12-cb816a30d79b" (UID: "5b1060ff-0459-4df0-9b12-cb816a30d79b"). InnerVolumeSpecName "kube-api-access-c7t58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:51:11.311321 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:11.311241 5041 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1060ff-0459-4df0-9b12-cb816a30d79b-service-ca-bundle\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:51:11.311321 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:11.311266 5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1060ff-0459-4df0-9b12-cb816a30d79b-trusted-ca-bundle\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:51:11.311321 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:11.311276 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7t58\" (UniqueName: \"kubernetes.io/projected/5b1060ff-0459-4df0-9b12-cb816a30d79b-kube-api-access-c7t58\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:51:11.311321 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:11.311285 5041 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5b1060ff-0459-4df0-9b12-cb816a30d79b-archives-path\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:51:12.015302 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:12.015270 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-s4cck-xtkdh" Nov 03 21:51:12.015302 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:12.015292 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-s4cck-xtkdh" event={"ID":"5b1060ff-0459-4df0-9b12-cb816a30d79b","Type":"ContainerDied","Data":"66e9b412329881719f5bb04fd760e70d98a327678e9d69ef26d216474c2b0521"} Nov 03 21:51:12.015525 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:12.015343 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66e9b412329881719f5bb04fd760e70d98a327678e9d69ef26d216474c2b0521" Nov 03 21:51:15.721269 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.721233 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6"] Nov 03 21:51:15.721985 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.721957 5041 cpu_manager.go:414] "containerMap: removing container" podUID="5b1060ff-0459-4df0-9b12-cb816a30d79b" containerName="insights-gathering" Nov 03 21:51:15.721985 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.721983 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1060ff-0459-4df0-9b12-cb816a30d79b" containerName="insights-gathering" Nov 03 21:51:15.722187 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.722032 5041 cpu_manager.go:414] "containerMap: removing container" podUID="c14b358c-04e7-42bf-b914-f41d424cad39" containerName="revoke-keys" Nov 03 21:51:15.722187 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.722045 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14b358c-04e7-42bf-b914-f41d424cad39" containerName="revoke-keys" Nov 03 21:51:15.722187 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.722166 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14b358c-04e7-42bf-b914-f41d424cad39" containerName="revoke-keys" Nov 03 21:51:15.722187 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.722184 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1060ff-0459-4df0-9b12-cb816a30d79b" containerName="insights-gathering" Nov 03 21:51:15.727003 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.726979 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" Nov 03 21:51:15.744414 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.744298 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6"] Nov 03 21:51:15.752426 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.752398 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:51:15.752615 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.752463 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:51:15.860943 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.860910 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlzrc\" (UniqueName: \"kubernetes.io/projected/3d9c173a-a699-4331-86f0-b821d90f79c5-kube-api-access-mlzrc\") pod \"maas-api-revoke-sub-ec843018-r8dn6\" (UID: \"3d9c173a-a699-4331-86f0-b821d90f79c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" Nov 03 21:51:15.861132 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.860979 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d9c173a-a699-4331-86f0-b821d90f79c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-r8dn6\" (UID: \"3d9c173a-a699-4331-86f0-b821d90f79c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" Nov 03 21:51:15.861132 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.861029 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d9c173a-a699-4331-86f0-b821d90f79c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-r8dn6\" (UID: \"3d9c173a-a699-4331-86f0-b821d90f79c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" Nov 03 21:51:15.976072 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.975221 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlzrc\" (UniqueName: \"kubernetes.io/projected/3d9c173a-a699-4331-86f0-b821d90f79c5-kube-api-access-mlzrc\") pod \"maas-api-revoke-sub-ec843018-r8dn6\" (UID: \"3d9c173a-a699-4331-86f0-b821d90f79c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" Nov 03 21:51:15.976072 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.975307 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d9c173a-a699-4331-86f0-b821d90f79c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-r8dn6\" (UID: \"3d9c173a-a699-4331-86f0-b821d90f79c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" Nov 03 21:51:15.976072 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.975334 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d9c173a-a699-4331-86f0-b821d90f79c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-r8dn6\" (UID: \"3d9c173a-a699-4331-86f0-b821d90f79c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" Nov 03 21:51:15.987480 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.977361 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d9c173a-a699-4331-86f0-b821d90f79c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-r8dn6\" (UID: \"3d9c173a-a699-4331-86f0-b821d90f79c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" Nov 03 21:51:15.987480 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.986507 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d9c173a-a699-4331-86f0-b821d90f79c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-r8dn6\" (UID: \"3d9c173a-a699-4331-86f0-b821d90f79c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" Nov 03 21:51:15.997521 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:15.997491 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlzrc\" (UniqueName: \"kubernetes.io/projected/3d9c173a-a699-4331-86f0-b821d90f79c5-kube-api-access-mlzrc\") pod \"maas-api-revoke-sub-ec843018-r8dn6\" (UID: \"3d9c173a-a699-4331-86f0-b821d90f79c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" Nov 03 21:51:16.045125 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:16.045081 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" Nov 03 21:51:16.349276 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:51:16.349239 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9c173a_a699_4331_86f0_b821d90f79c5.slice/crio-1b8461b7920fde9a4892593485011c604ba5313ba1bba9c81e4d097565c79c87 WatchSource:0}: Error finding container 1b8461b7920fde9a4892593485011c604ba5313ba1bba9c81e4d097565c79c87: Status 404 returned error can't find the container with id 1b8461b7920fde9a4892593485011c604ba5313ba1bba9c81e4d097565c79c87 Nov 03 21:51:16.349863 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:16.349771 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6"] Nov 03 21:51:17.065856 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:17.065813 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" event={"ID":"3d9c173a-a699-4331-86f0-b821d90f79c5","Type":"ContainerStarted","Data":"ffafd7776bddd435cb2b53fef61c56328fc95e20882ffb3aea2b2599be8973f4"} Nov 03 21:51:17.066339 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:17.065874 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" event={"ID":"3d9c173a-a699-4331-86f0-b821d90f79c5","Type":"ContainerStarted","Data":"1b8461b7920fde9a4892593485011c604ba5313ba1bba9c81e4d097565c79c87"} Nov 03 21:51:17.111617 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:17.111560 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" podStartSLOduration=2.111547275 podStartE2EDuration="2.111547275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:17.107614716 +0000 UTC m=+1292.614843023" watchObservedRunningTime="2026-11-03 21:51:17.111547275 +0000 UTC m=+1292.618775553" Nov 03 21:51:18.074613 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:18.074580 5041 generic.go:356] "Generic (PLEG): container finished" podID="3d9c173a-a699-4331-86f0-b821d90f79c5" containerID="ffafd7776bddd435cb2b53fef61c56328fc95e20882ffb3aea2b2599be8973f4" exitCode=0 Nov 03 21:51:18.075398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:18.074689 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" event={"ID":"3d9c173a-a699-4331-86f0-b821d90f79c5","Type":"ContainerDied","Data":"ffafd7776bddd435cb2b53fef61c56328fc95e20882ffb3aea2b2599be8973f4"} Nov 03 21:51:19.224860 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:19.224825 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" Nov 03 21:51:19.309203 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:19.309171 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d9c173a-a699-4331-86f0-b821d90f79c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d9c173a-a699-4331-86f0-b821d90f79c5-maas-api-openshift-ca\") pod \"3d9c173a-a699-4331-86f0-b821d90f79c5\" (UID: \"3d9c173a-a699-4331-86f0-b821d90f79c5\") " Nov 03 21:51:19.309389 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:19.309246 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9c173a-a699-4331-86f0-b821d90f79c5-kube-api-access-mlzrc\" (UniqueName: \"kubernetes.io/projected/3d9c173a-a699-4331-86f0-b821d90f79c5-kube-api-access-mlzrc\") pod \"3d9c173a-a699-4331-86f0-b821d90f79c5\" (UID: \"3d9c173a-a699-4331-86f0-b821d90f79c5\") " Nov 03 21:51:19.309389 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:19.309282 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d9c173a-a699-4331-86f0-b821d90f79c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d9c173a-a699-4331-86f0-b821d90f79c5-maas-api-xks-ca\") pod \"3d9c173a-a699-4331-86f0-b821d90f79c5\" (UID: \"3d9c173a-a699-4331-86f0-b821d90f79c5\") " Nov 03 21:51:19.309610 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:19.309587 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9c173a-a699-4331-86f0-b821d90f79c5-maas-api-openshift-ca" pod "3d9c173a-a699-4331-86f0-b821d90f79c5" (UID: "3d9c173a-a699-4331-86f0-b821d90f79c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:51:19.309769 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:19.309751 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d9c173a-a699-4331-86f0-b821d90f79c5-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:51:19.311361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:19.311333 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d9c173a-a699-4331-86f0-b821d90f79c5-maas-api-xks-ca" pod "3d9c173a-a699-4331-86f0-b821d90f79c5" (UID: "3d9c173a-a699-4331-86f0-b821d90f79c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:51:19.311460 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:19.311400 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9c173a-a699-4331-86f0-b821d90f79c5-kube-api-access-mlzrc" pod "3d9c173a-a699-4331-86f0-b821d90f79c5" (UID: "3d9c173a-a699-4331-86f0-b821d90f79c5"). InnerVolumeSpecName "kube-api-access-mlzrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:51:19.410583 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:19.410500 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlzrc\" (UniqueName: \"kubernetes.io/projected/3d9c173a-a699-4331-86f0-b821d90f79c5-kube-api-access-mlzrc\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:51:19.410583 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:19.410532 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d9c173a-a699-4331-86f0-b821d90f79c5-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:51:20.090899 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:20.090869 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" Nov 03 21:51:20.091081 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:20.090908 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6" event={"ID":"3d9c173a-a699-4331-86f0-b821d90f79c5","Type":"ContainerDied","Data":"1b8461b7920fde9a4892593485011c604ba5313ba1bba9c81e4d097565c79c87"} Nov 03 21:51:20.091081 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:20.090954 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b8461b7920fde9a4892593485011c604ba5313ba1bba9c81e4d097565c79c87" Nov 03 21:51:22.027565 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:22.027528 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59"] Nov 03 21:51:22.031599 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:22.031570 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1072f064-4dx59"] Nov 03 21:51:22.621479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:22.621437 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd0a1d5-d575-4352-b513-130a585dbeec" path="/var/lib/kubelet/pods/ffd0a1d5-d575-4352-b513-130a585dbeec/volumes" Nov 03 21:51:26.027339 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:26.027254 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98"] Nov 03 21:51:26.040622 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:26.040592 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10523d1-zns98"] Nov 03 21:51:26.620947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:26.620909 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="703148d0-66d5-40ab-844c-6299df569634" path="/var/lib/kubelet/pods/703148d0-66d5-40ab-844c-6299df569634/volumes" Nov 03 21:51:32.040041 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:32.039993 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj"] Nov 03 21:51:32.050965 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:32.050936 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8811c72-z9vcj"] Nov 03 21:51:32.625599 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:32.625559 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec01e33-6310-4936-8620-117f0ab603eb" path="/var/lib/kubelet/pods/7ec01e33-6310-4936-8620-117f0ab603eb/volumes" Nov 03 21:51:40.775977 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:51:40.775937 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:51:41.027680 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:41.027583 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh"] Nov 03 21:51:41.040238 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:41.040209 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da63b65f-bmlgh"] Nov 03 21:51:42.620327 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:42.620293 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fafa0ffa-bbbf-4a07-a651-6cbc1ff38939" path="/var/lib/kubelet/pods/fafa0ffa-bbbf-4a07-a651-6cbc1ff38939/volumes" Nov 03 21:51:48.020082 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:48.020043 5041 scope.go:122] "RemoveContainer" containerID="deb16faa31afc3dc89797176f21ace5376dce98c22f471a4546c43e96f0b42fa" Nov 03 21:51:48.030199 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:48.030177 5041 scope.go:122] "RemoveContainer" containerID="7ed16b6cd4703f556c23abb841135e54604090021fc1968728b64a9dffdeedb6" Nov 03 21:51:48.039284 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:48.039264 5041 scope.go:122] "RemoveContainer" containerID="d395c450f535d3c6d78fd035ea068dd829832fba790ebc6412c726559a5df867" Nov 03 21:51:49.262195 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.262158 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv"] Nov 03 21:51:49.262718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.262678 5041 cpu_manager.go:414] "containerMap: removing container" podUID="3d9c173a-a699-4331-86f0-b821d90f79c5" containerName="revoke-keys" Nov 03 21:51:49.262718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.262691 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d9c173a-a699-4331-86f0-b821d90f79c5" containerName="revoke-keys" Nov 03 21:51:49.262815 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.262794 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d9c173a-a699-4331-86f0-b821d90f79c5" containerName="revoke-keys" Nov 03 21:51:49.267917 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.267899 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" Nov 03 21:51:49.284603 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.284575 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv"] Nov 03 21:51:49.292325 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.292294 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:51:49.292462 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.292374 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:51:49.388198 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.388166 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-j4bcv\" (UID: \"7eaa0e47-926c-4621-82ce-d9a022c2cfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" Nov 03 21:51:49.388372 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.388268 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zscn4\" (UniqueName: \"kubernetes.io/projected/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-kube-api-access-zscn4\") pod \"maas-api-revoke-sub-046e1343-j4bcv\" (UID: \"7eaa0e47-926c-4621-82ce-d9a022c2cfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" Nov 03 21:51:49.388372 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.388291 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-j4bcv\" (UID: \"7eaa0e47-926c-4621-82ce-d9a022c2cfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" Nov 03 21:51:49.489332 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.489298 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-j4bcv\" (UID: \"7eaa0e47-926c-4621-82ce-d9a022c2cfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" Nov 03 21:51:49.489500 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.489384 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zscn4\" (UniqueName: \"kubernetes.io/projected/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-kube-api-access-zscn4\") pod \"maas-api-revoke-sub-046e1343-j4bcv\" (UID: \"7eaa0e47-926c-4621-82ce-d9a022c2cfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" Nov 03 21:51:49.489500 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.489408 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-j4bcv\" (UID: \"7eaa0e47-926c-4621-82ce-d9a022c2cfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" Nov 03 21:51:49.490010 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.489985 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-j4bcv\" (UID: \"7eaa0e47-926c-4621-82ce-d9a022c2cfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" Nov 03 21:51:49.491768 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.491744 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-j4bcv\" (UID: \"7eaa0e47-926c-4621-82ce-d9a022c2cfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" Nov 03 21:51:49.509551 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.509518 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zscn4\" (UniqueName: \"kubernetes.io/projected/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-kube-api-access-zscn4\") pod \"maas-api-revoke-sub-046e1343-j4bcv\" (UID: \"7eaa0e47-926c-4621-82ce-d9a022c2cfbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" Nov 03 21:51:49.583271 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.583190 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" Nov 03 21:51:49.878578 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:51:49.878543 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eaa0e47_926c_4621_82ce_d9a022c2cfbd.slice/crio-a1b51da193403b532e119507a9e68474afff91b3d284f77fd12c5aee63c47f51 WatchSource:0}: Error finding container a1b51da193403b532e119507a9e68474afff91b3d284f77fd12c5aee63c47f51: Status 404 returned error can't find the container with id a1b51da193403b532e119507a9e68474afff91b3d284f77fd12c5aee63c47f51 Nov 03 21:51:49.878753 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:49.878728 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv"] Nov 03 21:51:50.390744 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:50.390703 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" event={"ID":"7eaa0e47-926c-4621-82ce-d9a022c2cfbd","Type":"ContainerStarted","Data":"bc9c6a97d6152dd10a326343a5550c87cbb9a33d862f5ac55e04c669a4722598"} Nov 03 21:51:50.391211 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:50.390762 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" event={"ID":"7eaa0e47-926c-4621-82ce-d9a022c2cfbd","Type":"ContainerStarted","Data":"a1b51da193403b532e119507a9e68474afff91b3d284f77fd12c5aee63c47f51"} Nov 03 21:51:50.432904 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:50.432843 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" podStartSLOduration=1.432829234 podStartE2EDuration="1.432829234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:51:50.43149244 +0000 UTC m=+1325.938720718" watchObservedRunningTime="2026-11-03 21:51:50.432829234 +0000 UTC m=+1325.940057512" Nov 03 21:51:51.398783 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:51.398743 5041 generic.go:356] "Generic (PLEG): container finished" podID="7eaa0e47-926c-4621-82ce-d9a022c2cfbd" containerID="bc9c6a97d6152dd10a326343a5550c87cbb9a33d862f5ac55e04c669a4722598" exitCode=0 Nov 03 21:51:51.399172 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:51.398834 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" event={"ID":"7eaa0e47-926c-4621-82ce-d9a022c2cfbd","Type":"ContainerDied","Data":"bc9c6a97d6152dd10a326343a5550c87cbb9a33d862f5ac55e04c669a4722598"} Nov 03 21:51:52.548994 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:52.548964 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" Nov 03 21:51:52.723429 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:52.723402 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-maas-api-openshift-ca\") pod \"7eaa0e47-926c-4621-82ce-d9a022c2cfbd\" (UID: \"7eaa0e47-926c-4621-82ce-d9a022c2cfbd\") " Nov 03 21:51:52.723605 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:52.723442 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-maas-api-xks-ca\") pod \"7eaa0e47-926c-4621-82ce-d9a022c2cfbd\" (UID: \"7eaa0e47-926c-4621-82ce-d9a022c2cfbd\") " Nov 03 21:51:52.723605 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:52.723482 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-kube-api-access-zscn4\" (UniqueName: \"kubernetes.io/projected/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-kube-api-access-zscn4\") pod \"7eaa0e47-926c-4621-82ce-d9a022c2cfbd\" (UID: \"7eaa0e47-926c-4621-82ce-d9a022c2cfbd\") " Nov 03 21:51:52.723894 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:52.723867 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-maas-api-openshift-ca" pod "7eaa0e47-926c-4621-82ce-d9a022c2cfbd" (UID: "7eaa0e47-926c-4621-82ce-d9a022c2cfbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:51:52.725657 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:52.725633 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-kube-api-access-zscn4" pod "7eaa0e47-926c-4621-82ce-d9a022c2cfbd" (UID: "7eaa0e47-926c-4621-82ce-d9a022c2cfbd"). InnerVolumeSpecName "kube-api-access-zscn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:51:52.725760 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:52.725636 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-maas-api-xks-ca" pod "7eaa0e47-926c-4621-82ce-d9a022c2cfbd" (UID: "7eaa0e47-926c-4621-82ce-d9a022c2cfbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:51:52.825011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:52.824977 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:51:52.825011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:52.825008 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:51:52.825011 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:52.825017 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zscn4\" (UniqueName: \"kubernetes.io/projected/7eaa0e47-926c-4621-82ce-d9a022c2cfbd-kube-api-access-zscn4\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:51:53.413496 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:53.413456 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" Nov 03 21:51:53.413691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:53.413488 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv" event={"ID":"7eaa0e47-926c-4621-82ce-d9a022c2cfbd","Type":"ContainerDied","Data":"a1b51da193403b532e119507a9e68474afff91b3d284f77fd12c5aee63c47f51"} Nov 03 21:51:53.413691 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:53.413549 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1b51da193403b532e119507a9e68474afff91b3d284f77fd12c5aee63c47f51" Nov 03 21:51:59.028239 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.028205 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh"] Nov 03 21:51:59.041598 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.041568 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dceeffb-hwdxh"] Nov 03 21:51:59.606946 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.606911 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx"] Nov 03 21:51:59.607449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.607436 5041 cpu_manager.go:414] "containerMap: removing container" podUID="7eaa0e47-926c-4621-82ce-d9a022c2cfbd" containerName="revoke-keys" Nov 03 21:51:59.607449 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.607449 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eaa0e47-926c-4621-82ce-d9a022c2cfbd" containerName="revoke-keys" Nov 03 21:51:59.607540 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.607533 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eaa0e47-926c-4621-82ce-d9a022c2cfbd" containerName="revoke-keys" Nov 03 21:51:59.610882 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.610863 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx" Nov 03 21:51:59.633347 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.633308 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx"] Nov 03 21:51:59.638130 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.638102 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:51:59.638757 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.638730 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:51:59.686513 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.686482 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c98657-51d8-4b44-bf28-19279813860c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8l5lx\" (UID: \"d6c98657-51d8-4b44-bf28-19279813860c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx" Nov 03 21:51:59.686720 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.686544 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c98657-51d8-4b44-bf28-19279813860c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8l5lx\" (UID: \"d6c98657-51d8-4b44-bf28-19279813860c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx" Nov 03 21:51:59.686720 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.686563 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rp6w\" (UniqueName: \"kubernetes.io/projected/d6c98657-51d8-4b44-bf28-19279813860c-kube-api-access-2rp6w\") pod \"maas-api-revoke-sub-d3fba056-8l5lx\" (UID: \"d6c98657-51d8-4b44-bf28-19279813860c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx" Nov 03 21:51:59.787382 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.787345 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c98657-51d8-4b44-bf28-19279813860c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8l5lx\" (UID: \"d6c98657-51d8-4b44-bf28-19279813860c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx" Nov 03 21:51:59.787571 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.787411 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c98657-51d8-4b44-bf28-19279813860c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8l5lx\" (UID: \"d6c98657-51d8-4b44-bf28-19279813860c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx" Nov 03 21:51:59.787571 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.787438 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rp6w\" (UniqueName: \"kubernetes.io/projected/d6c98657-51d8-4b44-bf28-19279813860c-kube-api-access-2rp6w\") pod \"maas-api-revoke-sub-d3fba056-8l5lx\" (UID: \"d6c98657-51d8-4b44-bf28-19279813860c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx" Nov 03 21:51:59.788140 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.788113 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c98657-51d8-4b44-bf28-19279813860c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8l5lx\" (UID: \"d6c98657-51d8-4b44-bf28-19279813860c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx" Nov 03 21:51:59.789772 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.789748 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c98657-51d8-4b44-bf28-19279813860c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8l5lx\" (UID: \"d6c98657-51d8-4b44-bf28-19279813860c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx" Nov 03 21:51:59.807258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.807230 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rp6w\" (UniqueName: \"kubernetes.io/projected/d6c98657-51d8-4b44-bf28-19279813860c-kube-api-access-2rp6w\") pod \"maas-api-revoke-sub-d3fba056-8l5lx\" (UID: \"d6c98657-51d8-4b44-bf28-19279813860c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx" Nov 03 21:51:59.926381 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:51:59.926286 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx" Nov 03 21:52:00.035447 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:00.035410 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99"] Nov 03 21:52:00.041272 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:00.041238 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a99bb5-qcw99"] Nov 03 21:52:00.227161 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:00.227133 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx"] Nov 03 21:52:00.227530 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:52:00.227496 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c98657_51d8_4b44_bf28_19279813860c.slice/crio-a05ffffbab73c57e69e8b3a1d3c053b87a12279a3f1cd9a022c94c37a4c1df16 WatchSource:0}: Error finding container a05ffffbab73c57e69e8b3a1d3c053b87a12279a3f1cd9a022c94c37a4c1df16: Status 404 returned error can't find the container with id a05ffffbab73c57e69e8b3a1d3c053b87a12279a3f1cd9a022c94c37a4c1df16 Nov 03 21:52:00.464511 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:00.464472 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx" event={"ID":"d6c98657-51d8-4b44-bf28-19279813860c","Type":"ContainerStarted","Data":"a05ffffbab73c57e69e8b3a1d3c053b87a12279a3f1cd9a022c94c37a4c1df16"} Nov 03 21:52:00.621674 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:00.621628 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="454e2120-320f-4213-b2f5-0c0034a5f05d" path="/var/lib/kubelet/pods/454e2120-320f-4213-b2f5-0c0034a5f05d/volumes" Nov 03 21:52:00.622118 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:00.622102 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80847094-3884-43b5-8145-78f14994e864" path="/var/lib/kubelet/pods/80847094-3884-43b5-8145-78f14994e864/volumes" Nov 03 21:52:01.479645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:01.479603 5041 generic.go:356] "Generic (PLEG): container finished" podID="d6c98657-51d8-4b44-bf28-19279813860c" containerID="9d95f578692a5e826601696c61716cacea801aaaa2bdf6433bff4d54116063b3" exitCode=0 Nov 03 21:52:01.480169 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:01.479715 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx" event={"ID":"d6c98657-51d8-4b44-bf28-19279813860c","Type":"ContainerDied","Data":"9d95f578692a5e826601696c61716cacea801aaaa2bdf6433bff4d54116063b3"} Nov 03 21:52:03.338913 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:03.338863 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx" Nov 03 21:52:03.528052 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:03.528015 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx" event={"ID":"d6c98657-51d8-4b44-bf28-19279813860c","Type":"ContainerDied","Data":"a05ffffbab73c57e69e8b3a1d3c053b87a12279a3f1cd9a022c94c37a4c1df16"} Nov 03 21:52:03.528333 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:03.528315 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a05ffffbab73c57e69e8b3a1d3c053b87a12279a3f1cd9a022c94c37a4c1df16" Nov 03 21:52:03.528508 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:03.528492 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx" Nov 03 21:52:03.580055 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:03.580006 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c98657-51d8-4b44-bf28-19279813860c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c98657-51d8-4b44-bf28-19279813860c-maas-api-xks-ca\") pod \"d6c98657-51d8-4b44-bf28-19279813860c\" (UID: \"d6c98657-51d8-4b44-bf28-19279813860c\") " Nov 03 21:52:03.580372 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:03.580347 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c98657-51d8-4b44-bf28-19279813860c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c98657-51d8-4b44-bf28-19279813860c-maas-api-openshift-ca\") pod \"d6c98657-51d8-4b44-bf28-19279813860c\" (UID: \"d6c98657-51d8-4b44-bf28-19279813860c\") " Nov 03 21:52:03.580580 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:03.580559 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c98657-51d8-4b44-bf28-19279813860c-kube-api-access-2rp6w\" (UniqueName: \"kubernetes.io/projected/d6c98657-51d8-4b44-bf28-19279813860c-kube-api-access-2rp6w\") pod \"d6c98657-51d8-4b44-bf28-19279813860c\" (UID: \"d6c98657-51d8-4b44-bf28-19279813860c\") " Nov 03 21:52:03.582410 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:03.582385 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c98657-51d8-4b44-bf28-19279813860c-maas-api-openshift-ca" pod "d6c98657-51d8-4b44-bf28-19279813860c" (UID: "d6c98657-51d8-4b44-bf28-19279813860c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:52:03.591876 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:03.591848 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c98657-51d8-4b44-bf28-19279813860c-maas-api-xks-ca" pod "d6c98657-51d8-4b44-bf28-19279813860c" (UID: "d6c98657-51d8-4b44-bf28-19279813860c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:52:03.593852 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:03.593827 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c98657-51d8-4b44-bf28-19279813860c-kube-api-access-2rp6w" pod "d6c98657-51d8-4b44-bf28-19279813860c" (UID: "d6c98657-51d8-4b44-bf28-19279813860c"). InnerVolumeSpecName "kube-api-access-2rp6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:52:03.682386 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:03.682345 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c98657-51d8-4b44-bf28-19279813860c-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:52:03.682681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:03.682633 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c98657-51d8-4b44-bf28-19279813860c-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:52:03.682814 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:03.682798 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rp6w\" (UniqueName: \"kubernetes.io/projected/d6c98657-51d8-4b44-bf28-19279813860c-kube-api-access-2rp6w\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:52:10.027766 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:10.027724 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4"] Nov 03 21:52:10.041707 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:10.041677 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81b36a7b-q86r4"] Nov 03 21:52:10.620425 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:10.620378 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a26f5bfa-3760-40eb-b7cf-d5da5e18271a" path="/var/lib/kubelet/pods/a26f5bfa-3760-40eb-b7cf-d5da5e18271a/volumes" Nov 03 21:52:10.777545 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:52:10.777506 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:52:13.026159 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:13.026123 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd"] Nov 03 21:52:13.039835 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:13.039804 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fbcb306-qqffd"] Nov 03 21:52:14.621097 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:14.621052 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9cd6fb2-502f-4504-8d0d-b880dbcb20b9" path="/var/lib/kubelet/pods/d9cd6fb2-502f-4504-8d0d-b880dbcb20b9/volumes" Nov 03 21:52:15.027330 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:15.027294 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm"] Nov 03 21:52:15.040965 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:15.040931 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d768a4-q7pwm"] Nov 03 21:52:16.625432 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:16.625389 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0be0f94-8035-4087-be7b-1a988edecae5" path="/var/lib/kubelet/pods/d0be0f94-8035-4087-be7b-1a988edecae5/volumes" Nov 03 21:52:20.031459 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:20.031410 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484"] Nov 03 21:52:20.042645 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:20.042611 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5108f5c-4g484"] Nov 03 21:52:20.628185 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:20.628142 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e56751c-ebda-4b44-8967-10b0e7ecbc81" path="/var/lib/kubelet/pods/1e56751c-ebda-4b44-8967-10b0e7ecbc81/volumes" Nov 03 21:52:24.313801 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.313767 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq"] Nov 03 21:52:24.316209 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.314240 5041 cpu_manager.go:414] "containerMap: removing container" podUID="d6c98657-51d8-4b44-bf28-19279813860c" containerName="revoke-keys" Nov 03 21:52:24.316209 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.314250 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c98657-51d8-4b44-bf28-19279813860c" containerName="revoke-keys" Nov 03 21:52:24.316209 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.314343 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c98657-51d8-4b44-bf28-19279813860c" containerName="revoke-keys" Nov 03 21:52:24.317387 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.317372 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq" Nov 03 21:52:24.334980 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.334948 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq"] Nov 03 21:52:24.344826 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.344799 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:52:24.344979 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.344826 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:52:24.425380 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.425342 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rz5jq\" (UID: \"003f4ad4-0acd-4bdd-872e-bcbba93cbaad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq" Nov 03 21:52:24.425586 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.425404 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rz5jq\" (UID: \"003f4ad4-0acd-4bdd-872e-bcbba93cbaad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq" Nov 03 21:52:24.425586 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.425490 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xslh8\" (UniqueName: \"kubernetes.io/projected/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-kube-api-access-xslh8\") pod \"maas-api-revoke-sub-ae594979-rz5jq\" (UID: \"003f4ad4-0acd-4bdd-872e-bcbba93cbaad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq" Nov 03 21:52:24.526675 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.526632 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rz5jq\" (UID: \"003f4ad4-0acd-4bdd-872e-bcbba93cbaad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq" Nov 03 21:52:24.526889 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.526719 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rz5jq\" (UID: \"003f4ad4-0acd-4bdd-872e-bcbba93cbaad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq" Nov 03 21:52:24.526889 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.526737 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xslh8\" (UniqueName: \"kubernetes.io/projected/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-kube-api-access-xslh8\") pod \"maas-api-revoke-sub-ae594979-rz5jq\" (UID: \"003f4ad4-0acd-4bdd-872e-bcbba93cbaad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq" Nov 03 21:52:24.527377 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.527350 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rz5jq\" (UID: \"003f4ad4-0acd-4bdd-872e-bcbba93cbaad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq" Nov 03 21:52:24.529087 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.529061 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rz5jq\" (UID: \"003f4ad4-0acd-4bdd-872e-bcbba93cbaad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq" Nov 03 21:52:24.546963 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.546929 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xslh8\" (UniqueName: \"kubernetes.io/projected/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-kube-api-access-xslh8\") pod \"maas-api-revoke-sub-ae594979-rz5jq\" (UID: \"003f4ad4-0acd-4bdd-872e-bcbba93cbaad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq" Nov 03 21:52:24.630292 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.630218 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq" Nov 03 21:52:24.928952 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:24.928922 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq"] Nov 03 21:52:24.929296 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:52:24.929273 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003f4ad4_0acd_4bdd_872e_bcbba93cbaad.slice/crio-840d34c98655eff68894e423f7ea76011f293251587b974691df3eb9e1c7a6d8 WatchSource:0}: Error finding container 840d34c98655eff68894e423f7ea76011f293251587b974691df3eb9e1c7a6d8: Status 404 returned error can't find the container with id 840d34c98655eff68894e423f7ea76011f293251587b974691df3eb9e1c7a6d8 Nov 03 21:52:25.034241 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:25.034171 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr"] Nov 03 21:52:25.036923 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:25.036900 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98a75a46-pjccr"] Nov 03 21:52:25.746690 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:25.746647 5041 generic.go:356] "Generic (PLEG): container finished" podID="003f4ad4-0acd-4bdd-872e-bcbba93cbaad" containerID="4d4ad06c046f010b130ce21672f7c036a33f91ca0b072f0e36b0697d5c1032ad" exitCode=0 Nov 03 21:52:25.747113 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:25.746734 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq" event={"ID":"003f4ad4-0acd-4bdd-872e-bcbba93cbaad","Type":"ContainerDied","Data":"4d4ad06c046f010b130ce21672f7c036a33f91ca0b072f0e36b0697d5c1032ad"} Nov 03 21:52:25.747113 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:25.746837 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq" event={"ID":"003f4ad4-0acd-4bdd-872e-bcbba93cbaad","Type":"ContainerStarted","Data":"840d34c98655eff68894e423f7ea76011f293251587b974691df3eb9e1c7a6d8"} Nov 03 21:52:26.620966 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:26.620933 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19298766-3255-421f-98f8-f90cd92e55f3" path="/var/lib/kubelet/pods/19298766-3255-421f-98f8-f90cd92e55f3/volumes" Nov 03 21:52:26.894832 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:26.894802 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq" Nov 03 21:52:26.953164 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:26.953131 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-maas-api-xks-ca\") pod \"003f4ad4-0acd-4bdd-872e-bcbba93cbaad\" (UID: \"003f4ad4-0acd-4bdd-872e-bcbba93cbaad\") " Nov 03 21:52:26.953339 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:26.953269 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-kube-api-access-xslh8\" (UniqueName: \"kubernetes.io/projected/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-kube-api-access-xslh8\") pod \"003f4ad4-0acd-4bdd-872e-bcbba93cbaad\" (UID: \"003f4ad4-0acd-4bdd-872e-bcbba93cbaad\") " Nov 03 21:52:26.953339 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:26.953314 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-maas-api-openshift-ca\") pod \"003f4ad4-0acd-4bdd-872e-bcbba93cbaad\" (UID: \"003f4ad4-0acd-4bdd-872e-bcbba93cbaad\") " Nov 03 21:52:26.953727 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:26.953698 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-maas-api-openshift-ca" pod "003f4ad4-0acd-4bdd-872e-bcbba93cbaad" (UID: "003f4ad4-0acd-4bdd-872e-bcbba93cbaad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:52:26.955277 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:26.955255 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-maas-api-xks-ca" pod "003f4ad4-0acd-4bdd-872e-bcbba93cbaad" (UID: "003f4ad4-0acd-4bdd-872e-bcbba93cbaad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:52:26.955712 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:26.955683 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-kube-api-access-xslh8" pod "003f4ad4-0acd-4bdd-872e-bcbba93cbaad" (UID: "003f4ad4-0acd-4bdd-872e-bcbba93cbaad"). InnerVolumeSpecName "kube-api-access-xslh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:52:27.054763 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:27.054729 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xslh8\" (UniqueName: \"kubernetes.io/projected/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-kube-api-access-xslh8\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:52:27.054763 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:27.054758 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:52:27.054763 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:27.054768 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003f4ad4-0acd-4bdd-872e-bcbba93cbaad-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:52:27.762999 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:27.762971 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq" Nov 03 21:52:27.763179 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:27.763009 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq" event={"ID":"003f4ad4-0acd-4bdd-872e-bcbba93cbaad","Type":"ContainerDied","Data":"840d34c98655eff68894e423f7ea76011f293251587b974691df3eb9e1c7a6d8"} Nov 03 21:52:27.763179 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:27.763057 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="840d34c98655eff68894e423f7ea76011f293251587b974691df3eb9e1c7a6d8" Nov 03 21:52:35.026358 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:35.026316 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh"] Nov 03 21:52:35.039738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:35.039708 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33efaec-wrlsh"] Nov 03 21:52:36.027773 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:36.027743 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6"] Nov 03 21:52:36.040928 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:36.040886 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2"] Nov 03 21:52:36.044131 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:36.044110 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c4bdca-57zs6"] Nov 03 21:52:36.047316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:36.047291 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da877d2-7lqb2"] Nov 03 21:52:36.621470 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:36.621426 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b91308d-036f-4fcb-9df5-1572f9821a2c" path="/var/lib/kubelet/pods/1b91308d-036f-4fcb-9df5-1572f9821a2c/volumes" Nov 03 21:52:36.621984 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:36.621961 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb90896c-0b0c-4685-b2d4-207407cb4959" path="/var/lib/kubelet/pods/eb90896c-0b0c-4685-b2d4-207407cb4959/volumes" Nov 03 21:52:36.622422 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:36.622401 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb92ba06-6d13-4b60-a133-565d7c821595" path="/var/lib/kubelet/pods/fb92ba06-6d13-4b60-a133-565d7c821595/volumes" Nov 03 21:52:37.027216 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:37.027185 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b"] Nov 03 21:52:37.040289 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:37.040259 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-321bd940-2qf7b"] Nov 03 21:52:38.625311 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:38.625262 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb2992a-df1d-4f5c-befd-b1d352bb79dc" path="/var/lib/kubelet/pods/cfb2992a-df1d-4f5c-befd-b1d352bb79dc/volumes" Nov 03 21:52:40.897426 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:52:40.897383 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:52:41.043218 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:41.043177 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255"] Nov 03 21:52:41.056984 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:41.056936 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e905ad-l4255"] Nov 03 21:52:42.621040 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:42.621002 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a96e07b-e342-4628-a091-00528b5d6d1c" path="/var/lib/kubelet/pods/5a96e07b-e342-4628-a091-00528b5d6d1c/volumes" Nov 03 21:52:44.916238 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:44.916206 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t"] Nov 03 21:52:44.916721 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:44.916706 5041 cpu_manager.go:414] "containerMap: removing container" podUID="003f4ad4-0acd-4bdd-872e-bcbba93cbaad" containerName="revoke-keys" Nov 03 21:52:44.916784 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:44.916722 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003f4ad4-0acd-4bdd-872e-bcbba93cbaad" containerName="revoke-keys" Nov 03 21:52:44.916834 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:44.916823 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="003f4ad4-0acd-4bdd-872e-bcbba93cbaad" containerName="revoke-keys" Nov 03 21:52:44.921619 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:44.921603 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t" Nov 03 21:52:44.937317 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:44.937290 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t"] Nov 03 21:52:44.947153 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:44.947126 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:52:44.947282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:44.947200 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:52:45.119133 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:45.119091 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7z4q\" (UniqueName: \"kubernetes.io/projected/2aded170-bd02-42f0-90fc-0fcf113d8fe0-kube-api-access-w7z4q\") pod \"maas-api-revoke-sub-a4aef7d7-9dn9t\" (UID: \"2aded170-bd02-42f0-90fc-0fcf113d8fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t" Nov 03 21:52:45.119302 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:45.119239 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aded170-bd02-42f0-90fc-0fcf113d8fe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9dn9t\" (UID: \"2aded170-bd02-42f0-90fc-0fcf113d8fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t" Nov 03 21:52:45.119356 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:45.119326 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aded170-bd02-42f0-90fc-0fcf113d8fe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9dn9t\" (UID: \"2aded170-bd02-42f0-90fc-0fcf113d8fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t" Nov 03 21:52:45.219963 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:45.219927 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aded170-bd02-42f0-90fc-0fcf113d8fe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9dn9t\" (UID: \"2aded170-bd02-42f0-90fc-0fcf113d8fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t" Nov 03 21:52:45.220154 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:45.219989 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aded170-bd02-42f0-90fc-0fcf113d8fe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9dn9t\" (UID: \"2aded170-bd02-42f0-90fc-0fcf113d8fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t" Nov 03 21:52:45.220154 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:45.220021 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7z4q\" (UniqueName: \"kubernetes.io/projected/2aded170-bd02-42f0-90fc-0fcf113d8fe0-kube-api-access-w7z4q\") pod \"maas-api-revoke-sub-a4aef7d7-9dn9t\" (UID: \"2aded170-bd02-42f0-90fc-0fcf113d8fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t" Nov 03 21:52:45.220567 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:45.220541 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aded170-bd02-42f0-90fc-0fcf113d8fe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9dn9t\" (UID: \"2aded170-bd02-42f0-90fc-0fcf113d8fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t" Nov 03 21:52:45.222377 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:45.222351 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aded170-bd02-42f0-90fc-0fcf113d8fe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9dn9t\" (UID: \"2aded170-bd02-42f0-90fc-0fcf113d8fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t" Nov 03 21:52:45.240217 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:45.240191 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7z4q\" (UniqueName: \"kubernetes.io/projected/2aded170-bd02-42f0-90fc-0fcf113d8fe0-kube-api-access-w7z4q\") pod \"maas-api-revoke-sub-a4aef7d7-9dn9t\" (UID: \"2aded170-bd02-42f0-90fc-0fcf113d8fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t" Nov 03 21:52:45.535629 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:45.535543 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t" Nov 03 21:52:45.832526 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:52:45.832490 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aded170_bd02_42f0_90fc_0fcf113d8fe0.slice/crio-78de95aef47e8d2363d35443c2bc8663577158408a465b40f980d9c44b9d260c WatchSource:0}: Error finding container 78de95aef47e8d2363d35443c2bc8663577158408a465b40f980d9c44b9d260c: Status 404 returned error can't find the container with id 78de95aef47e8d2363d35443c2bc8663577158408a465b40f980d9c44b9d260c Nov 03 21:52:45.833308 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:45.833280 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t"] Nov 03 21:52:45.957353 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:45.957320 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t" event={"ID":"2aded170-bd02-42f0-90fc-0fcf113d8fe0","Type":"ContainerStarted","Data":"78de95aef47e8d2363d35443c2bc8663577158408a465b40f980d9c44b9d260c"} Nov 03 21:52:46.966253 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:46.966219 5041 generic.go:356] "Generic (PLEG): container finished" podID="2aded170-bd02-42f0-90fc-0fcf113d8fe0" containerID="e2cba7a2c76b6595f417f9bcf05cc90e5ac80cdb9496cbc89572ec27ea0b9e41" exitCode=0 Nov 03 21:52:46.966616 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:46.966299 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t" event={"ID":"2aded170-bd02-42f0-90fc-0fcf113d8fe0","Type":"ContainerDied","Data":"e2cba7a2c76b6595f417f9bcf05cc90e5ac80cdb9496cbc89572ec27ea0b9e41"} Nov 03 21:52:48.082353 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.082315 5041 scope.go:122] "RemoveContainer" containerID="d1028a2766cedaa9928cd8b849b3290ddf6cd579992b6c24299c497fdd2d8c42" Nov 03 21:52:48.093978 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.093954 5041 scope.go:122] "RemoveContainer" containerID="246b0103072bd417285fa7be281166b859deb7792cfd9b556a563cf82917bdf8" Nov 03 21:52:48.103627 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.103604 5041 scope.go:122] "RemoveContainer" containerID="d2587b4dd0fd47ef23d48b30df1bf0a07d494e9451beb3fa77b73fa71aa1ed35" Nov 03 21:52:48.132179 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.132158 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t" Nov 03 21:52:48.134061 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.134042 5041 scope.go:122] "RemoveContainer" containerID="62da3297e8bed3fc3eb32eaefd3b06307fcf9673b8f9c837fd8d834c8b59e2b5" Nov 03 21:52:48.146396 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.146371 5041 scope.go:122] "RemoveContainer" containerID="4f843bce1feae0f551fe66d8d2fdf55ad9929a640eecf6ccfd53aa1d711cf32c" Nov 03 21:52:48.146638 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.146617 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aded170-bd02-42f0-90fc-0fcf113d8fe0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aded170-bd02-42f0-90fc-0fcf113d8fe0-maas-api-openshift-ca\") pod \"2aded170-bd02-42f0-90fc-0fcf113d8fe0\" (UID: \"2aded170-bd02-42f0-90fc-0fcf113d8fe0\") " Nov 03 21:52:48.147076 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.146649 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aded170-bd02-42f0-90fc-0fcf113d8fe0-kube-api-access-w7z4q\" (UniqueName: \"kubernetes.io/projected/2aded170-bd02-42f0-90fc-0fcf113d8fe0-kube-api-access-w7z4q\") pod \"2aded170-bd02-42f0-90fc-0fcf113d8fe0\" (UID: \"2aded170-bd02-42f0-90fc-0fcf113d8fe0\") " Nov 03 21:52:48.147076 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.146685 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aded170-bd02-42f0-90fc-0fcf113d8fe0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aded170-bd02-42f0-90fc-0fcf113d8fe0-maas-api-xks-ca\") pod \"2aded170-bd02-42f0-90fc-0fcf113d8fe0\" (UID: \"2aded170-bd02-42f0-90fc-0fcf113d8fe0\") " Nov 03 21:52:48.147167 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.147106 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aded170-bd02-42f0-90fc-0fcf113d8fe0-maas-api-openshift-ca" pod "2aded170-bd02-42f0-90fc-0fcf113d8fe0" (UID: "2aded170-bd02-42f0-90fc-0fcf113d8fe0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:52:48.148859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.148836 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aded170-bd02-42f0-90fc-0fcf113d8fe0-maas-api-xks-ca" pod "2aded170-bd02-42f0-90fc-0fcf113d8fe0" (UID: "2aded170-bd02-42f0-90fc-0fcf113d8fe0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:52:48.148985 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.148966 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aded170-bd02-42f0-90fc-0fcf113d8fe0-kube-api-access-w7z4q" pod "2aded170-bd02-42f0-90fc-0fcf113d8fe0" (UID: "2aded170-bd02-42f0-90fc-0fcf113d8fe0"). InnerVolumeSpecName "kube-api-access-w7z4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:52:48.248323 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.248257 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aded170-bd02-42f0-90fc-0fcf113d8fe0-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:52:48.248323 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.248287 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7z4q\" (UniqueName: \"kubernetes.io/projected/2aded170-bd02-42f0-90fc-0fcf113d8fe0-kube-api-access-w7z4q\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:52:48.248323 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.248300 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aded170-bd02-42f0-90fc-0fcf113d8fe0-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:52:48.983974 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.983937 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t" event={"ID":"2aded170-bd02-42f0-90fc-0fcf113d8fe0","Type":"ContainerDied","Data":"78de95aef47e8d2363d35443c2bc8663577158408a465b40f980d9c44b9d260c"} Nov 03 21:52:48.983974 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.983964 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t" Nov 03 21:52:48.984235 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:48.983987 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78de95aef47e8d2363d35443c2bc8663577158408a465b40f980d9c44b9d260c" Nov 03 21:52:51.030086 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:51.030052 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt"] Nov 03 21:52:51.043444 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:51.043415 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cb49f77-sdqdt"] Nov 03 21:52:52.620531 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:52:52.620492 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a88155b-e3ff-4308-94ee-d8cea2d3ed6a" path="/var/lib/kubelet/pods/5a88155b-e3ff-4308-94ee-d8cea2d3ed6a/volumes" Nov 03 21:53:08.033576 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.033497 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz"] Nov 03 21:53:08.047349 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.047309 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6"] Nov 03 21:53:08.050464 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.050430 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d"] Nov 03 21:53:08.053486 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.053452 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5290ae2-zmmpz"] Nov 03 21:53:08.056126 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.056102 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6587b599-x8xn6"] Nov 03 21:53:08.059725 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.059686 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bdf0ab2-9zr4d"] Nov 03 21:53:08.110333 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.110300 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw"] Nov 03 21:53:08.110792 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.110779 5041 cpu_manager.go:414] "containerMap: removing container" podUID="2aded170-bd02-42f0-90fc-0fcf113d8fe0" containerName="revoke-keys" Nov 03 21:53:08.110842 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.110793 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aded170-bd02-42f0-90fc-0fcf113d8fe0" containerName="revoke-keys" Nov 03 21:53:08.110898 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.110889 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aded170-bd02-42f0-90fc-0fcf113d8fe0" containerName="revoke-keys" Nov 03 21:53:08.114251 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.114235 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" Nov 03 21:53:08.130319 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.130295 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:53:08.130467 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.130394 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:53:08.132135 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.132112 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw"] Nov 03 21:53:08.181571 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.181542 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0476f3-a816-494a-b1bb-1da86667bf9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6bztw\" (UID: \"cb0476f3-a816-494a-b1bb-1da86667bf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" Nov 03 21:53:08.181817 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.181579 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7dfc\" (UniqueName: \"kubernetes.io/projected/cb0476f3-a816-494a-b1bb-1da86667bf9c-kube-api-access-d7dfc\") pod \"maas-api-revoke-sub-79805ca9-6bztw\" (UID: \"cb0476f3-a816-494a-b1bb-1da86667bf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" Nov 03 21:53:08.181817 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.181748 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0476f3-a816-494a-b1bb-1da86667bf9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6bztw\" (UID: \"cb0476f3-a816-494a-b1bb-1da86667bf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" Nov 03 21:53:08.282317 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.282274 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0476f3-a816-494a-b1bb-1da86667bf9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6bztw\" (UID: \"cb0476f3-a816-494a-b1bb-1da86667bf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" Nov 03 21:53:08.282317 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.282322 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dfc\" (UniqueName: \"kubernetes.io/projected/cb0476f3-a816-494a-b1bb-1da86667bf9c-kube-api-access-d7dfc\") pod \"maas-api-revoke-sub-79805ca9-6bztw\" (UID: \"cb0476f3-a816-494a-b1bb-1da86667bf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" Nov 03 21:53:08.282539 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.282382 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0476f3-a816-494a-b1bb-1da86667bf9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6bztw\" (UID: \"cb0476f3-a816-494a-b1bb-1da86667bf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" Nov 03 21:53:08.283282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.283247 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0476f3-a816-494a-b1bb-1da86667bf9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6bztw\" (UID: \"cb0476f3-a816-494a-b1bb-1da86667bf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" Nov 03 21:53:08.284623 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.284581 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0476f3-a816-494a-b1bb-1da86667bf9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6bztw\" (UID: \"cb0476f3-a816-494a-b1bb-1da86667bf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" Nov 03 21:53:08.302808 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.302778 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7dfc\" (UniqueName: \"kubernetes.io/projected/cb0476f3-a816-494a-b1bb-1da86667bf9c-kube-api-access-d7dfc\") pod \"maas-api-revoke-sub-79805ca9-6bztw\" (UID: \"cb0476f3-a816-494a-b1bb-1da86667bf9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" Nov 03 21:53:08.427498 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.427462 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" Nov 03 21:53:08.620718 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.620630 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f38640-6dc3-47b9-8e3b-36efdbd57c68" path="/var/lib/kubelet/pods/56f38640-6dc3-47b9-8e3b-36efdbd57c68/volumes" Nov 03 21:53:08.621066 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.621052 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76d6f982-faed-4a04-aa25-85a2159c343b" path="/var/lib/kubelet/pods/76d6f982-faed-4a04-aa25-85a2159c343b/volumes" Nov 03 21:53:08.621428 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.621415 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93" path="/var/lib/kubelet/pods/d8f1aa57-cc39-4b87-9fce-ab8fb8e6be93/volumes" Nov 03 21:53:08.725310 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:08.725284 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw"] Nov 03 21:53:08.726440 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:53:08.726415 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0476f3_a816_494a_b1bb_1da86667bf9c.slice/crio-b0f1405243c5c608ad93a338a7ab80990b0986c6b2c32f856d84d22cb046b7e2 WatchSource:0}: Error finding container b0f1405243c5c608ad93a338a7ab80990b0986c6b2c32f856d84d22cb046b7e2: Status 404 returned error can't find the container with id b0f1405243c5c608ad93a338a7ab80990b0986c6b2c32f856d84d22cb046b7e2 Nov 03 21:53:09.150735 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:09.150704 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" event={"ID":"cb0476f3-a816-494a-b1bb-1da86667bf9c","Type":"ContainerStarted","Data":"cecc0efc81c2887b9501d2894372ad82aebba46ef45b037437b2fae271a35ede"} Nov 03 21:53:09.151141 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:09.150758 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" event={"ID":"cb0476f3-a816-494a-b1bb-1da86667bf9c","Type":"ContainerStarted","Data":"b0f1405243c5c608ad93a338a7ab80990b0986c6b2c32f856d84d22cb046b7e2"} Nov 03 21:53:09.193841 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:09.193756 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" podStartSLOduration=1.193742866 podStartE2EDuration="1.193742866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:53:09.192218173 +0000 UTC m=+1404.699446479" watchObservedRunningTime="2026-11-03 21:53:09.193742866 +0000 UTC m=+1404.700971170" Nov 03 21:53:10.159431 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:10.159391 5041 generic.go:356] "Generic (PLEG): container finished" podID="cb0476f3-a816-494a-b1bb-1da86667bf9c" containerID="cecc0efc81c2887b9501d2894372ad82aebba46ef45b037437b2fae271a35ede" exitCode=0 Nov 03 21:53:10.159863 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:10.159459 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" event={"ID":"cb0476f3-a816-494a-b1bb-1da86667bf9c","Type":"ContainerDied","Data":"cecc0efc81c2887b9501d2894372ad82aebba46ef45b037437b2fae271a35ede"} Nov 03 21:53:10.777068 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:53:10.777034 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:53:11.029037 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:11.028952 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql"] Nov 03 21:53:11.042070 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:11.042033 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1561a7cd-gstql"] Nov 03 21:53:11.312789 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:11.312760 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" Nov 03 21:53:11.406504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:11.406478 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb0476f3-a816-494a-b1bb-1da86667bf9c-kube-api-access-d7dfc\" (UniqueName: \"kubernetes.io/projected/cb0476f3-a816-494a-b1bb-1da86667bf9c-kube-api-access-d7dfc\") pod \"cb0476f3-a816-494a-b1bb-1da86667bf9c\" (UID: \"cb0476f3-a816-494a-b1bb-1da86667bf9c\") " Nov 03 21:53:11.406703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:11.406539 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb0476f3-a816-494a-b1bb-1da86667bf9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0476f3-a816-494a-b1bb-1da86667bf9c-maas-api-xks-ca\") pod \"cb0476f3-a816-494a-b1bb-1da86667bf9c\" (UID: \"cb0476f3-a816-494a-b1bb-1da86667bf9c\") " Nov 03 21:53:11.406703 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:11.406627 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb0476f3-a816-494a-b1bb-1da86667bf9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0476f3-a816-494a-b1bb-1da86667bf9c-maas-api-openshift-ca\") pod \"cb0476f3-a816-494a-b1bb-1da86667bf9c\" (UID: \"cb0476f3-a816-494a-b1bb-1da86667bf9c\") " Nov 03 21:53:11.406969 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:11.406943 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0476f3-a816-494a-b1bb-1da86667bf9c-maas-api-openshift-ca" pod "cb0476f3-a816-494a-b1bb-1da86667bf9c" (UID: "cb0476f3-a816-494a-b1bb-1da86667bf9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:53:11.408694 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:11.408650 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb0476f3-a816-494a-b1bb-1da86667bf9c-maas-api-xks-ca" pod "cb0476f3-a816-494a-b1bb-1da86667bf9c" (UID: "cb0476f3-a816-494a-b1bb-1da86667bf9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:53:11.408786 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:11.408655 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0476f3-a816-494a-b1bb-1da86667bf9c-kube-api-access-d7dfc" pod "cb0476f3-a816-494a-b1bb-1da86667bf9c" (UID: "cb0476f3-a816-494a-b1bb-1da86667bf9c"). InnerVolumeSpecName "kube-api-access-d7dfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:53:11.507461 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:11.507431 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0476f3-a816-494a-b1bb-1da86667bf9c-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:53:11.507461 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:11.507458 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0476f3-a816-494a-b1bb-1da86667bf9c-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:53:11.507461 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:11.507468 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7dfc\" (UniqueName: \"kubernetes.io/projected/cb0476f3-a816-494a-b1bb-1da86667bf9c-kube-api-access-d7dfc\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:53:12.174431 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:12.174397 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" Nov 03 21:53:12.174617 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:12.174426 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw" event={"ID":"cb0476f3-a816-494a-b1bb-1da86667bf9c","Type":"ContainerDied","Data":"b0f1405243c5c608ad93a338a7ab80990b0986c6b2c32f856d84d22cb046b7e2"} Nov 03 21:53:12.174617 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:12.174485 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0f1405243c5c608ad93a338a7ab80990b0986c6b2c32f856d84d22cb046b7e2" Nov 03 21:53:12.621145 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:12.621107 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a8a6d82-56a3-4ef0-9785-f9578579b5f0" path="/var/lib/kubelet/pods/3a8a6d82-56a3-4ef0-9785-f9578579b5f0/volumes" Nov 03 21:53:14.028894 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:14.028857 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt"] Nov 03 21:53:14.043066 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:14.043030 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4"] Nov 03 21:53:14.046070 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:14.046044 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c787ced1-7tcjt"] Nov 03 21:53:14.048760 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:14.048740 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c21b054-zq8l4"] Nov 03 21:53:14.621631 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:14.621594 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff7c5c1-0631-4596-ac51-a0578f81953c" path="/var/lib/kubelet/pods/3ff7c5c1-0631-4596-ac51-a0578f81953c/volumes" Nov 03 21:53:14.622030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:14.622014 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44581ba9-5165-4cbd-91ea-3f53777faf74" path="/var/lib/kubelet/pods/44581ba9-5165-4cbd-91ea-3f53777faf74/volumes" Nov 03 21:53:22.027067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:22.027029 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg"] Nov 03 21:53:22.040728 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:22.040698 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-885ff216-96mdg"] Nov 03 21:53:22.625544 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:22.625491 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a894e07-8a02-471c-90f6-0dc4db7c25e9" path="/var/lib/kubelet/pods/8a894e07-8a02-471c-90f6-0dc4db7c25e9/volumes" Nov 03 21:53:23.028828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:23.028782 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z"] Nov 03 21:53:23.043752 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:23.043709 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f80204f-h952z"] Nov 03 21:53:24.590318 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:24.589786 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6kgrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:53:24.590318 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:24.589853 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" podUID="bd662fa8-196b-4cad-b414-1bc2dbd68aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:53:24.643279 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:24.643241 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d975a235-5acc-47f6-9fc1-da6131f27b1b" path="/var/lib/kubelet/pods/d975a235-5acc-47f6-9fc1-da6131f27b1b/volumes" Nov 03 21:53:28.194043 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.194001 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr"] Nov 03 21:53:28.194808 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.194790 5041 cpu_manager.go:414] "containerMap: removing container" podUID="cb0476f3-a816-494a-b1bb-1da86667bf9c" containerName="revoke-keys" Nov 03 21:53:28.194875 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.194811 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb0476f3-a816-494a-b1bb-1da86667bf9c" containerName="revoke-keys" Nov 03 21:53:28.194977 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.194966 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb0476f3-a816-494a-b1bb-1da86667bf9c" containerName="revoke-keys" Nov 03 21:53:28.200434 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.200406 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" Nov 03 21:53:28.213971 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.213934 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr"] Nov 03 21:53:28.222558 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.222528 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:53:28.222711 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.222557 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:53:28.330588 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.330550 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sx9cr\" (UID: \"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" Nov 03 21:53:28.330797 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.330773 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sx9cr\" (UID: \"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" Nov 03 21:53:28.330849 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.330821 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm72f\" (UniqueName: \"kubernetes.io/projected/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-kube-api-access-gm72f\") pod \"maas-api-revoke-sub-0b7e8edb-sx9cr\" (UID: \"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" Nov 03 21:53:28.432010 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.431944 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sx9cr\" (UID: \"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" Nov 03 21:53:28.432194 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.432160 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sx9cr\" (UID: \"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" Nov 03 21:53:28.432246 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.432212 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm72f\" (UniqueName: \"kubernetes.io/projected/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-kube-api-access-gm72f\") pod \"maas-api-revoke-sub-0b7e8edb-sx9cr\" (UID: \"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" Nov 03 21:53:28.432815 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.432789 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sx9cr\" (UID: \"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" Nov 03 21:53:28.434431 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.434402 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sx9cr\" (UID: \"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" Nov 03 21:53:28.452770 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.452743 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm72f\" (UniqueName: \"kubernetes.io/projected/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-kube-api-access-gm72f\") pod \"maas-api-revoke-sub-0b7e8edb-sx9cr\" (UID: \"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" Nov 03 21:53:28.520175 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.520142 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" Nov 03 21:53:28.820165 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:28.820133 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr"] Nov 03 21:53:28.820886 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:53:28.820859 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4cc25f7_b147_42ee_8e76_1d124f2cbc8e.slice/crio-284fb2a588920d534ce08a4d0aec0e4c7bd242c826499636a6f6860ea19bc910 WatchSource:0}: Error finding container 284fb2a588920d534ce08a4d0aec0e4c7bd242c826499636a6f6860ea19bc910: Status 404 returned error can't find the container with id 284fb2a588920d534ce08a4d0aec0e4c7bd242c826499636a6f6860ea19bc910 Nov 03 21:53:29.455418 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:29.455382 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" event={"ID":"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e","Type":"ContainerStarted","Data":"deb5ab0a57ce3b3f0c3c9ed504cf9e577c8da79a164b3f0c6a9a89e66a1b4c72"} Nov 03 21:53:29.455846 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:29.455438 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" event={"ID":"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e","Type":"ContainerStarted","Data":"284fb2a588920d534ce08a4d0aec0e4c7bd242c826499636a6f6860ea19bc910"} Nov 03 21:53:29.498063 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:29.498007 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" podStartSLOduration=1.497992598 podStartE2EDuration="1.497992598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:53:29.495012923 +0000 UTC m=+1425.002241203" watchObservedRunningTime="2026-11-03 21:53:29.497992598 +0000 UTC m=+1425.005220877" Nov 03 21:53:30.464270 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:30.464230 5041 generic.go:356] "Generic (PLEG): container finished" podID="c4cc25f7-b147-42ee-8e76-1d124f2cbc8e" containerID="deb5ab0a57ce3b3f0c3c9ed504cf9e577c8da79a164b3f0c6a9a89e66a1b4c72" exitCode=0 Nov 03 21:53:30.464762 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:30.464319 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" event={"ID":"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e","Type":"ContainerDied","Data":"deb5ab0a57ce3b3f0c3c9ed504cf9e577c8da79a164b3f0c6a9a89e66a1b4c72"} Nov 03 21:53:31.626368 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:31.626335 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" Nov 03 21:53:31.768868 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:31.768765 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-kube-api-access-gm72f\" (UniqueName: \"kubernetes.io/projected/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-kube-api-access-gm72f\") pod \"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e\" (UID: \"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e\") " Nov 03 21:53:31.768868 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:31.768818 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-maas-api-openshift-ca\") pod \"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e\" (UID: \"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e\") " Nov 03 21:53:31.769138 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:31.768895 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-maas-api-xks-ca\") pod \"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e\" (UID: \"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e\") " Nov 03 21:53:31.769297 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:31.769259 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-maas-api-openshift-ca" pod "c4cc25f7-b147-42ee-8e76-1d124f2cbc8e" (UID: "c4cc25f7-b147-42ee-8e76-1d124f2cbc8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:53:31.771184 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:31.771148 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-maas-api-xks-ca" pod "c4cc25f7-b147-42ee-8e76-1d124f2cbc8e" (UID: "c4cc25f7-b147-42ee-8e76-1d124f2cbc8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:53:31.771184 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:31.771158 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-kube-api-access-gm72f" pod "c4cc25f7-b147-42ee-8e76-1d124f2cbc8e" (UID: "c4cc25f7-b147-42ee-8e76-1d124f2cbc8e"). InnerVolumeSpecName "kube-api-access-gm72f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:53:31.869995 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:31.869951 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm72f\" (UniqueName: \"kubernetes.io/projected/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-kube-api-access-gm72f\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:53:31.869995 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:31.869990 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:53:31.869995 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:31.870000 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:53:32.481123 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:32.481095 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" Nov 03 21:53:32.481298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:32.481091 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr" event={"ID":"c4cc25f7-b147-42ee-8e76-1d124f2cbc8e","Type":"ContainerDied","Data":"284fb2a588920d534ce08a4d0aec0e4c7bd242c826499636a6f6860ea19bc910"} Nov 03 21:53:32.481298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:32.481218 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="284fb2a588920d534ce08a4d0aec0e4c7bd242c826499636a6f6860ea19bc910" Nov 03 21:53:40.778037 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:53:40.777997 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:53:48.218002 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:48.217963 5041 scope.go:122] "RemoveContainer" containerID="27878dbe06d0d2c03b121a5ac2534098b7f52b21d860fab1e532c8b6cb132d7f" Nov 03 21:53:48.228604 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:48.228580 5041 scope.go:122] "RemoveContainer" containerID="113a7f4bd8f4b1d9651b8f6b520edae1403f05d70118a93ff10506a61c7eec5a" Nov 03 21:53:48.238454 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:48.238431 5041 scope.go:122] "RemoveContainer" containerID="6cba811328011ece107f3f8f1a989a87bfe8a1c13276029f25f6eb2d3ec88a6d" Nov 03 21:53:48.248471 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:48.248438 5041 scope.go:122] "RemoveContainer" containerID="d3c1e7a091849f80152306c3ff905974091b4571276ad13761b0688061087ba6" Nov 03 21:53:48.258266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:48.258244 5041 scope.go:122] "RemoveContainer" containerID="af415b5d0ea5e8a7bccc79960ba63611e16ab419f510044ed4d041ecfdde14b9" Nov 03 21:53:48.267747 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:48.267723 5041 scope.go:122] "RemoveContainer" containerID="7710cb7fd75d70a1178353af87156ef858f2bc30797e2cd0bb468d6fee5af142" Nov 03 21:53:48.277738 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:48.277719 5041 scope.go:122] "RemoveContainer" containerID="55e82961f73dcd4d75d535456e6fff75dde722ab588c78deb4ef060f75d51155" Nov 03 21:53:48.289538 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:48.289518 5041 scope.go:122] "RemoveContainer" containerID="7de9237a0a0e27a0945e40d6e059832d9b354de855dbbc621a22c1794c110399" Nov 03 21:53:48.299177 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:48.299158 5041 scope.go:122] "RemoveContainer" containerID="5866060ddbf7f3a07faf44cabe0a62466a74dd92cb97a8c7393b163372d23dd7" Nov 03 21:53:48.308425 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:48.308405 5041 scope.go:122] "RemoveContainer" containerID="72a0a9e334c3748df5e8de15f723eda542a364d013137ffdbebb357a4f1175c8" Nov 03 21:53:48.317630 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:48.317609 5041 scope.go:122] "RemoveContainer" containerID="3ce6087af1c70a9297e913f3b5b89c4b4af39ff751e74d6da144f99ec3977458" Nov 03 21:53:48.329772 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:48.329747 5041 scope.go:122] "RemoveContainer" containerID="2181969444fd8d1757b3c1fc627d9cfa40e433d65e7f97637e2f9803f8287224" Nov 03 21:53:48.339069 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:48.339046 5041 scope.go:122] "RemoveContainer" containerID="7f79c34a4d0217dbfeaf3b85a472c32074f0c6ca7a3381167450a0d45df63b7c" Nov 03 21:53:48.348150 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:48.348130 5041 scope.go:122] "RemoveContainer" containerID="81c57b9719cfdc2e4abd53351a9f876070cff142eb5913fdf35474adccb72efd" Nov 03 21:53:48.357097 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:48.357077 5041 scope.go:122] "RemoveContainer" containerID="332c0e231e5c8a2ddd59ebe3bcecc3856d3735662a93c86468039c61c50c45f2" Nov 03 21:53:48.365985 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:48.365966 5041 scope.go:122] "RemoveContainer" containerID="680b5d6f97f0d63266206268f29eb6927c6bea1155b4d6d843b0ad80f7a1fc1e" Nov 03 21:53:49.042942 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.042901 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj"] Nov 03 21:53:49.043406 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.043391 5041 cpu_manager.go:414] "containerMap: removing container" podUID="c4cc25f7-b147-42ee-8e76-1d124f2cbc8e" containerName="revoke-keys" Nov 03 21:53:49.043406 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.043406 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4cc25f7-b147-42ee-8e76-1d124f2cbc8e" containerName="revoke-keys" Nov 03 21:53:49.043561 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.043503 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4cc25f7-b147-42ee-8e76-1d124f2cbc8e" containerName="revoke-keys" Nov 03 21:53:49.050201 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.050173 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj" Nov 03 21:53:49.067935 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.067900 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj"] Nov 03 21:53:49.074376 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.074341 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:53:49.078058 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.077981 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:53:49.147928 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.147889 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb8kk\" (UniqueName: \"kubernetes.io/projected/be401861-774a-44dc-bffd-e4cba87a8918-kube-api-access-rb8kk\") pod \"maas-api-revoke-sub-3d4156f1-hnnbj\" (UID: \"be401861-774a-44dc-bffd-e4cba87a8918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj" Nov 03 21:53:49.148114 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.147951 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be401861-774a-44dc-bffd-e4cba87a8918-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hnnbj\" (UID: \"be401861-774a-44dc-bffd-e4cba87a8918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj" Nov 03 21:53:49.148114 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.148059 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be401861-774a-44dc-bffd-e4cba87a8918-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hnnbj\" (UID: \"be401861-774a-44dc-bffd-e4cba87a8918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj" Nov 03 21:53:49.249522 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.249485 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8kk\" (UniqueName: \"kubernetes.io/projected/be401861-774a-44dc-bffd-e4cba87a8918-kube-api-access-rb8kk\") pod \"maas-api-revoke-sub-3d4156f1-hnnbj\" (UID: \"be401861-774a-44dc-bffd-e4cba87a8918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj" Nov 03 21:53:49.249995 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.249537 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be401861-774a-44dc-bffd-e4cba87a8918-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hnnbj\" (UID: \"be401861-774a-44dc-bffd-e4cba87a8918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj" Nov 03 21:53:49.249995 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.249575 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be401861-774a-44dc-bffd-e4cba87a8918-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hnnbj\" (UID: \"be401861-774a-44dc-bffd-e4cba87a8918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj" Nov 03 21:53:49.250380 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.250351 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be401861-774a-44dc-bffd-e4cba87a8918-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hnnbj\" (UID: \"be401861-774a-44dc-bffd-e4cba87a8918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj" Nov 03 21:53:49.252026 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.252003 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be401861-774a-44dc-bffd-e4cba87a8918-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hnnbj\" (UID: \"be401861-774a-44dc-bffd-e4cba87a8918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj" Nov 03 21:53:49.271930 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.271897 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8kk\" (UniqueName: \"kubernetes.io/projected/be401861-774a-44dc-bffd-e4cba87a8918-kube-api-access-rb8kk\") pod \"maas-api-revoke-sub-3d4156f1-hnnbj\" (UID: \"be401861-774a-44dc-bffd-e4cba87a8918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj" Nov 03 21:53:49.380087 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.380002 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj" Nov 03 21:53:49.678949 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:49.678914 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj"] Nov 03 21:53:49.679280 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:53:49.679258 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe401861_774a_44dc_bffd_e4cba87a8918.slice/crio-63b9c2999d7265939b442635a7352c7d320c96760717cfeec7984c1da4f02ddd WatchSource:0}: Error finding container 63b9c2999d7265939b442635a7352c7d320c96760717cfeec7984c1da4f02ddd: Status 404 returned error can't find the container with id 63b9c2999d7265939b442635a7352c7d320c96760717cfeec7984c1da4f02ddd Nov 03 21:53:50.627190 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:50.627152 5041 generic.go:356] "Generic (PLEG): container finished" podID="be401861-774a-44dc-bffd-e4cba87a8918" containerID="89a36ab5a3babf05141d9026cdc0c754d15c97e8b708136a8e0e2af18f4dc3c5" exitCode=0 Nov 03 21:53:50.627595 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:50.627198 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj" event={"ID":"be401861-774a-44dc-bffd-e4cba87a8918","Type":"ContainerDied","Data":"89a36ab5a3babf05141d9026cdc0c754d15c97e8b708136a8e0e2af18f4dc3c5"} Nov 03 21:53:50.627595 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:50.627246 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj" event={"ID":"be401861-774a-44dc-bffd-e4cba87a8918","Type":"ContainerStarted","Data":"63b9c2999d7265939b442635a7352c7d320c96760717cfeec7984c1da4f02ddd"} Nov 03 21:53:51.783802 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:51.783774 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj" Nov 03 21:53:51.877200 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:51.877167 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be401861-774a-44dc-bffd-e4cba87a8918-kube-api-access-rb8kk\" (UniqueName: \"kubernetes.io/projected/be401861-774a-44dc-bffd-e4cba87a8918-kube-api-access-rb8kk\") pod \"be401861-774a-44dc-bffd-e4cba87a8918\" (UID: \"be401861-774a-44dc-bffd-e4cba87a8918\") " Nov 03 21:53:51.877385 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:51.877218 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be401861-774a-44dc-bffd-e4cba87a8918-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be401861-774a-44dc-bffd-e4cba87a8918-maas-api-openshift-ca\") pod \"be401861-774a-44dc-bffd-e4cba87a8918\" (UID: \"be401861-774a-44dc-bffd-e4cba87a8918\") " Nov 03 21:53:51.877385 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:51.877361 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be401861-774a-44dc-bffd-e4cba87a8918-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be401861-774a-44dc-bffd-e4cba87a8918-maas-api-xks-ca\") pod \"be401861-774a-44dc-bffd-e4cba87a8918\" (UID: \"be401861-774a-44dc-bffd-e4cba87a8918\") " Nov 03 21:53:51.877569 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:51.877549 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be401861-774a-44dc-bffd-e4cba87a8918-maas-api-openshift-ca" pod "be401861-774a-44dc-bffd-e4cba87a8918" (UID: "be401861-774a-44dc-bffd-e4cba87a8918"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:53:51.877776 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:51.877755 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be401861-774a-44dc-bffd-e4cba87a8918-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:53:51.879437 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:51.879404 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be401861-774a-44dc-bffd-e4cba87a8918-maas-api-xks-ca" pod "be401861-774a-44dc-bffd-e4cba87a8918" (UID: "be401861-774a-44dc-bffd-e4cba87a8918"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:53:51.879437 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:51.879422 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be401861-774a-44dc-bffd-e4cba87a8918-kube-api-access-rb8kk" pod "be401861-774a-44dc-bffd-e4cba87a8918" (UID: "be401861-774a-44dc-bffd-e4cba87a8918"). InnerVolumeSpecName "kube-api-access-rb8kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:53:51.978779 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:51.978742 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb8kk\" (UniqueName: \"kubernetes.io/projected/be401861-774a-44dc-bffd-e4cba87a8918-kube-api-access-rb8kk\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:53:51.978779 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:51.978773 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be401861-774a-44dc-bffd-e4cba87a8918-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:53:52.641621 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:52.641591 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj" Nov 03 21:53:52.641780 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:52.641626 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj" event={"ID":"be401861-774a-44dc-bffd-e4cba87a8918","Type":"ContainerDied","Data":"63b9c2999d7265939b442635a7352c7d320c96760717cfeec7984c1da4f02ddd"} Nov 03 21:53:52.641780 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:52.641688 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63b9c2999d7265939b442635a7352c7d320c96760717cfeec7984c1da4f02ddd" Nov 03 21:53:59.032772 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:59.032736 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5"] Nov 03 21:53:59.046227 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:53:59.046192 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65bf3fab-9bkz5"] Nov 03 21:54:00.622111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:00.622067 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5150c7e7-763a-4b01-b493-d4046712ff6d" path="/var/lib/kubelet/pods/5150c7e7-763a-4b01-b493-d4046712ff6d/volumes" Nov 03 21:54:10.778520 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:54:10.778478 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:54:12.427217 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.427173 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj"] Nov 03 21:54:12.440954 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.440915 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hnnbj"] Nov 03 21:54:12.473027 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.472993 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz"] Nov 03 21:54:12.473494 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.473480 5041 cpu_manager.go:414] "containerMap: removing container" podUID="be401861-774a-44dc-bffd-e4cba87a8918" containerName="revoke-keys" Nov 03 21:54:12.473538 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.473496 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be401861-774a-44dc-bffd-e4cba87a8918" containerName="revoke-keys" Nov 03 21:54:12.473595 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.473587 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="be401861-774a-44dc-bffd-e4cba87a8918" containerName="revoke-keys" Nov 03 21:54:12.476902 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.476881 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" Nov 03 21:54:12.493079 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.493035 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:54:12.493079 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.493069 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:54:12.494205 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.494183 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz"] Nov 03 21:54:12.621346 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.621303 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be401861-774a-44dc-bffd-e4cba87a8918" path="/var/lib/kubelet/pods/be401861-774a-44dc-bffd-e4cba87a8918/volumes" Nov 03 21:54:12.626509 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.626483 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f63df7-0aab-44cc-9703-a0613ad73195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsvpz\" (UID: \"a5f63df7-0aab-44cc-9703-a0613ad73195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" Nov 03 21:54:12.626706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.626518 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwnqw\" (UniqueName: \"kubernetes.io/projected/a5f63df7-0aab-44cc-9703-a0613ad73195-kube-api-access-lwnqw\") pod \"maas-api-revoke-sub-3d4156f1-dsvpz\" (UID: \"a5f63df7-0aab-44cc-9703-a0613ad73195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" Nov 03 21:54:12.626706 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.626644 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f63df7-0aab-44cc-9703-a0613ad73195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsvpz\" (UID: \"a5f63df7-0aab-44cc-9703-a0613ad73195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" Nov 03 21:54:12.728170 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.728132 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f63df7-0aab-44cc-9703-a0613ad73195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsvpz\" (UID: \"a5f63df7-0aab-44cc-9703-a0613ad73195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" Nov 03 21:54:12.728362 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.728227 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f63df7-0aab-44cc-9703-a0613ad73195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsvpz\" (UID: \"a5f63df7-0aab-44cc-9703-a0613ad73195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" Nov 03 21:54:12.728362 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.728250 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwnqw\" (UniqueName: \"kubernetes.io/projected/a5f63df7-0aab-44cc-9703-a0613ad73195-kube-api-access-lwnqw\") pod \"maas-api-revoke-sub-3d4156f1-dsvpz\" (UID: \"a5f63df7-0aab-44cc-9703-a0613ad73195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" Nov 03 21:54:12.728880 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.728844 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f63df7-0aab-44cc-9703-a0613ad73195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsvpz\" (UID: \"a5f63df7-0aab-44cc-9703-a0613ad73195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" Nov 03 21:54:12.730695 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.730639 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f63df7-0aab-44cc-9703-a0613ad73195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dsvpz\" (UID: \"a5f63df7-0aab-44cc-9703-a0613ad73195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" Nov 03 21:54:12.748451 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.748427 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwnqw\" (UniqueName: \"kubernetes.io/projected/a5f63df7-0aab-44cc-9703-a0613ad73195-kube-api-access-lwnqw\") pod \"maas-api-revoke-sub-3d4156f1-dsvpz\" (UID: \"a5f63df7-0aab-44cc-9703-a0613ad73195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" Nov 03 21:54:12.794332 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:12.794299 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" Nov 03 21:54:13.095581 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:13.095549 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz"] Nov 03 21:54:13.096422 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:54:13.096379 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f63df7_0aab_44cc_9703_a0613ad73195.slice/crio-e2cfd8d7d09d970604a07e43e9d8571b6182b1deb77b22906c4fe9cdaf4d0829 WatchSource:0}: Error finding container e2cfd8d7d09d970604a07e43e9d8571b6182b1deb77b22906c4fe9cdaf4d0829: Status 404 returned error can't find the container with id e2cfd8d7d09d970604a07e43e9d8571b6182b1deb77b22906c4fe9cdaf4d0829 Nov 03 21:54:13.838800 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:13.838759 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" event={"ID":"a5f63df7-0aab-44cc-9703-a0613ad73195","Type":"ContainerStarted","Data":"2ffaa4f6f965b5242143279944cfbbef9e0db4bb9335117d231e6db0450eca05"} Nov 03 21:54:13.839211 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:13.838821 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" event={"ID":"a5f63df7-0aab-44cc-9703-a0613ad73195","Type":"ContainerStarted","Data":"e2cfd8d7d09d970604a07e43e9d8571b6182b1deb77b22906c4fe9cdaf4d0829"} Nov 03 21:54:13.882196 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:13.882140 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" podStartSLOduration=1.8821259399999999 podStartE2EDuration="1.88212594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:54:13.879175534 +0000 UTC m=+1469.386403815" watchObservedRunningTime="2026-11-03 21:54:13.88212594 +0000 UTC m=+1469.389354217" Nov 03 21:54:14.847405 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:14.847363 5041 generic.go:356] "Generic (PLEG): container finished" podID="a5f63df7-0aab-44cc-9703-a0613ad73195" containerID="2ffaa4f6f965b5242143279944cfbbef9e0db4bb9335117d231e6db0450eca05" exitCode=0 Nov 03 21:54:14.847405 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:14.847396 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" event={"ID":"a5f63df7-0aab-44cc-9703-a0613ad73195","Type":"ContainerDied","Data":"2ffaa4f6f965b5242143279944cfbbef9e0db4bb9335117d231e6db0450eca05"} Nov 03 21:54:16.003940 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:16.003915 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" Nov 03 21:54:16.164846 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:16.164763 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f63df7-0aab-44cc-9703-a0613ad73195-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f63df7-0aab-44cc-9703-a0613ad73195-maas-api-openshift-ca\") pod \"a5f63df7-0aab-44cc-9703-a0613ad73195\" (UID: \"a5f63df7-0aab-44cc-9703-a0613ad73195\") " Nov 03 21:54:16.165004 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:16.164873 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5f63df7-0aab-44cc-9703-a0613ad73195-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f63df7-0aab-44cc-9703-a0613ad73195-maas-api-xks-ca\") pod \"a5f63df7-0aab-44cc-9703-a0613ad73195\" (UID: \"a5f63df7-0aab-44cc-9703-a0613ad73195\") " Nov 03 21:54:16.165004 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:16.164949 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f63df7-0aab-44cc-9703-a0613ad73195-kube-api-access-lwnqw\" (UniqueName: \"kubernetes.io/projected/a5f63df7-0aab-44cc-9703-a0613ad73195-kube-api-access-lwnqw\") pod \"a5f63df7-0aab-44cc-9703-a0613ad73195\" (UID: \"a5f63df7-0aab-44cc-9703-a0613ad73195\") " Nov 03 21:54:16.165155 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:16.165127 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f63df7-0aab-44cc-9703-a0613ad73195-maas-api-openshift-ca" pod "a5f63df7-0aab-44cc-9703-a0613ad73195" (UID: "a5f63df7-0aab-44cc-9703-a0613ad73195"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:54:16.166989 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:16.166961 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f63df7-0aab-44cc-9703-a0613ad73195-kube-api-access-lwnqw" pod "a5f63df7-0aab-44cc-9703-a0613ad73195" (UID: "a5f63df7-0aab-44cc-9703-a0613ad73195"). InnerVolumeSpecName "kube-api-access-lwnqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:54:16.167098 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:16.167025 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f63df7-0aab-44cc-9703-a0613ad73195-maas-api-xks-ca" pod "a5f63df7-0aab-44cc-9703-a0613ad73195" (UID: "a5f63df7-0aab-44cc-9703-a0613ad73195"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:54:16.266496 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:16.266462 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwnqw\" (UniqueName: \"kubernetes.io/projected/a5f63df7-0aab-44cc-9703-a0613ad73195-kube-api-access-lwnqw\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:54:16.266496 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:16.266490 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f63df7-0aab-44cc-9703-a0613ad73195-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:54:16.266496 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:16.266499 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f63df7-0aab-44cc-9703-a0613ad73195-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:54:16.862556 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:16.862516 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" event={"ID":"a5f63df7-0aab-44cc-9703-a0613ad73195","Type":"ContainerDied","Data":"e2cfd8d7d09d970604a07e43e9d8571b6182b1deb77b22906c4fe9cdaf4d0829"} Nov 03 21:54:16.862556 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:16.862535 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz" Nov 03 21:54:16.862769 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:16.862565 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2cfd8d7d09d970604a07e43e9d8571b6182b1deb77b22906c4fe9cdaf4d0829" Nov 03 21:54:40.790938 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:54:40.790832 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:54:45.696566 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:45.696532 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 21:54:45.699411 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:45.699386 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 21:54:46.794707 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.794643 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6"] Nov 03 21:54:46.795203 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.795188 5041 cpu_manager.go:414] "containerMap: removing container" podUID="a5f63df7-0aab-44cc-9703-a0613ad73195" containerName="revoke-keys" Nov 03 21:54:46.795255 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.795204 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f63df7-0aab-44cc-9703-a0613ad73195" containerName="revoke-keys" Nov 03 21:54:46.795295 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.795287 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f63df7-0aab-44cc-9703-a0613ad73195" containerName="revoke-keys" Nov 03 21:54:46.797602 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.797581 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" Nov 03 21:54:46.816418 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.816389 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6"] Nov 03 21:54:46.817365 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.817342 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:54:46.825337 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.825314 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:54:46.853445 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.853411 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191390f3-d9fb-4471-99ee-a569ef09dca5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wdbm6\" (UID: \"191390f3-d9fb-4471-99ee-a569ef09dca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" Nov 03 21:54:46.853625 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.853475 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191390f3-d9fb-4471-99ee-a569ef09dca5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wdbm6\" (UID: \"191390f3-d9fb-4471-99ee-a569ef09dca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" Nov 03 21:54:46.853625 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.853590 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbwr2\" (UniqueName: \"kubernetes.io/projected/191390f3-d9fb-4471-99ee-a569ef09dca5-kube-api-access-sbwr2\") pod \"maas-api-revoke-sub-e2fc9c5b-wdbm6\" (UID: \"191390f3-d9fb-4471-99ee-a569ef09dca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" Nov 03 21:54:46.954852 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.954821 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191390f3-d9fb-4471-99ee-a569ef09dca5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wdbm6\" (UID: \"191390f3-d9fb-4471-99ee-a569ef09dca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" Nov 03 21:54:46.955034 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.954865 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191390f3-d9fb-4471-99ee-a569ef09dca5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wdbm6\" (UID: \"191390f3-d9fb-4471-99ee-a569ef09dca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" Nov 03 21:54:46.955082 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.955043 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbwr2\" (UniqueName: \"kubernetes.io/projected/191390f3-d9fb-4471-99ee-a569ef09dca5-kube-api-access-sbwr2\") pod \"maas-api-revoke-sub-e2fc9c5b-wdbm6\" (UID: \"191390f3-d9fb-4471-99ee-a569ef09dca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" Nov 03 21:54:46.955456 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.955436 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191390f3-d9fb-4471-99ee-a569ef09dca5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wdbm6\" (UID: \"191390f3-d9fb-4471-99ee-a569ef09dca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" Nov 03 21:54:46.957284 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.957263 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191390f3-d9fb-4471-99ee-a569ef09dca5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-wdbm6\" (UID: \"191390f3-d9fb-4471-99ee-a569ef09dca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" Nov 03 21:54:46.975746 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:46.975722 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbwr2\" (UniqueName: \"kubernetes.io/projected/191390f3-d9fb-4471-99ee-a569ef09dca5-kube-api-access-sbwr2\") pod \"maas-api-revoke-sub-e2fc9c5b-wdbm6\" (UID: \"191390f3-d9fb-4471-99ee-a569ef09dca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" Nov 03 21:54:47.027575 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:47.027543 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8"] Nov 03 21:54:47.032298 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:47.032260 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8lkg8"] Nov 03 21:54:47.111403 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:47.111327 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" Nov 03 21:54:47.410717 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:47.410690 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6"] Nov 03 21:54:47.411902 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:54:47.411873 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191390f3_d9fb_4471_99ee_a569ef09dca5.slice/crio-abdb85e11cbfe5048417b56905c8dbcba8480f7e6776f04168ed7caaa5861dc9 WatchSource:0}: Error finding container abdb85e11cbfe5048417b56905c8dbcba8480f7e6776f04168ed7caaa5861dc9: Status 404 returned error can't find the container with id abdb85e11cbfe5048417b56905c8dbcba8480f7e6776f04168ed7caaa5861dc9 Nov 03 21:54:48.138760 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:48.138645 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" event={"ID":"191390f3-d9fb-4471-99ee-a569ef09dca5","Type":"ContainerStarted","Data":"a3819e362a8aa9c847718bc8babe0e4d41b8e3a6a796dd74c9098bd05bfb2b4b"} Nov 03 21:54:48.138760 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:48.138729 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" event={"ID":"191390f3-d9fb-4471-99ee-a569ef09dca5","Type":"ContainerStarted","Data":"abdb85e11cbfe5048417b56905c8dbcba8480f7e6776f04168ed7caaa5861dc9"} Nov 03 21:54:48.179462 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:48.179410 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" podStartSLOduration=2.179395518 podStartE2EDuration="2.179395518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:54:48.178545271 +0000 UTC m=+1503.685773571" watchObservedRunningTime="2026-11-03 21:54:48.179395518 +0000 UTC m=+1503.686623800" Nov 03 21:54:48.464278 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:48.464249 5041 scope.go:122] "RemoveContainer" containerID="9da7a13156bff0c32521416b328df37c79ce771b02d6d0419c34ddc85d1a8860" Nov 03 21:54:48.474233 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:48.474214 5041 scope.go:122] "RemoveContainer" containerID="d4acb8a9d294b980b519a5b62cb67f8ef3ee45e583e4746a15ce01a1914ad428" Nov 03 21:54:48.483336 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:48.483314 5041 scope.go:122] "RemoveContainer" containerID="431a999d092bf3a8241e74ea0d26e7821eaccae7f3fdb424fa08bf2488724704" Nov 03 21:54:48.492580 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:48.492558 5041 scope.go:122] "RemoveContainer" containerID="f4df0a13f0e35259e11caaee6d9e5369d876cee1ef22b5f7b7088d0cf278bdac" Nov 03 21:54:48.501813 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:48.501792 5041 scope.go:122] "RemoveContainer" containerID="acb75c4939541f7360323c73831a67c3b88d7a8484d9f753bbbdf23012b1037e" Nov 03 21:54:48.511523 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:48.511503 5041 scope.go:122] "RemoveContainer" containerID="e8257daa8319f30855856d96d1aa014d1caea78bb4e7446f109516c46daa93df" Nov 03 21:54:48.520137 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:48.520113 5041 scope.go:122] "RemoveContainer" containerID="aad69d224ec50e56097f4f228828f66374683b9a84103357860a1589219fc2f8" Nov 03 21:54:48.528879 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:48.528858 5041 scope.go:122] "RemoveContainer" containerID="16747b27e60ebc8489a94b02ab0ad34196a5e53d2752b1bfe39ba19b7dac509b" Nov 03 21:54:48.537178 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:48.537162 5041 scope.go:122] "RemoveContainer" containerID="38763417e5116db28b61ea25b76a4b1d8a9a833ef7829c45edffe7bdcf65f749" Nov 03 21:54:48.545364 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:48.545340 5041 scope.go:122] "RemoveContainer" containerID="51fd43d50e100e6c571479549e5ff91985c07f4a2af0e62a18e9daac92dd309c" Nov 03 21:54:48.620427 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:48.620399 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b" path="/var/lib/kubelet/pods/9e0ce3c5-169f-46c8-a3b7-6b2f61ddb68b/volumes" Nov 03 21:54:49.146774 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:49.146737 5041 generic.go:356] "Generic (PLEG): container finished" podID="191390f3-d9fb-4471-99ee-a569ef09dca5" containerID="a3819e362a8aa9c847718bc8babe0e4d41b8e3a6a796dd74c9098bd05bfb2b4b" exitCode=0 Nov 03 21:54:49.147160 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:49.146819 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" event={"ID":"191390f3-d9fb-4471-99ee-a569ef09dca5","Type":"ContainerDied","Data":"a3819e362a8aa9c847718bc8babe0e4d41b8e3a6a796dd74c9098bd05bfb2b4b"} Nov 03 21:54:50.303057 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:50.303031 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" Nov 03 21:54:50.388310 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:50.388274 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191390f3-d9fb-4471-99ee-a569ef09dca5-kube-api-access-sbwr2\" (UniqueName: \"kubernetes.io/projected/191390f3-d9fb-4471-99ee-a569ef09dca5-kube-api-access-sbwr2\") pod \"191390f3-d9fb-4471-99ee-a569ef09dca5\" (UID: \"191390f3-d9fb-4471-99ee-a569ef09dca5\") " Nov 03 21:54:50.388493 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:50.388420 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/191390f3-d9fb-4471-99ee-a569ef09dca5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191390f3-d9fb-4471-99ee-a569ef09dca5-maas-api-xks-ca\") pod \"191390f3-d9fb-4471-99ee-a569ef09dca5\" (UID: \"191390f3-d9fb-4471-99ee-a569ef09dca5\") " Nov 03 21:54:50.388493 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:50.388451 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/191390f3-d9fb-4471-99ee-a569ef09dca5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191390f3-d9fb-4471-99ee-a569ef09dca5-maas-api-openshift-ca\") pod \"191390f3-d9fb-4471-99ee-a569ef09dca5\" (UID: \"191390f3-d9fb-4471-99ee-a569ef09dca5\") " Nov 03 21:54:50.388901 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:50.388875 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191390f3-d9fb-4471-99ee-a569ef09dca5-maas-api-openshift-ca" pod "191390f3-d9fb-4471-99ee-a569ef09dca5" (UID: "191390f3-d9fb-4471-99ee-a569ef09dca5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:54:50.390504 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:50.390482 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191390f3-d9fb-4471-99ee-a569ef09dca5-maas-api-xks-ca" pod "191390f3-d9fb-4471-99ee-a569ef09dca5" (UID: "191390f3-d9fb-4471-99ee-a569ef09dca5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:54:50.390564 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:50.390524 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191390f3-d9fb-4471-99ee-a569ef09dca5-kube-api-access-sbwr2" pod "191390f3-d9fb-4471-99ee-a569ef09dca5" (UID: "191390f3-d9fb-4471-99ee-a569ef09dca5"). InnerVolumeSpecName "kube-api-access-sbwr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:54:50.489230 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:50.489199 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191390f3-d9fb-4471-99ee-a569ef09dca5-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:54:50.489230 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:50.489229 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191390f3-d9fb-4471-99ee-a569ef09dca5-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:54:50.489423 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:50.489240 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbwr2\" (UniqueName: \"kubernetes.io/projected/191390f3-d9fb-4471-99ee-a569ef09dca5-kube-api-access-sbwr2\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:54:51.160399 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:51.160368 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" Nov 03 21:54:51.160616 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:51.160362 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6" event={"ID":"191390f3-d9fb-4471-99ee-a569ef09dca5","Type":"ContainerDied","Data":"abdb85e11cbfe5048417b56905c8dbcba8480f7e6776f04168ed7caaa5861dc9"} Nov 03 21:54:51.160616 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:51.160496 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abdb85e11cbfe5048417b56905c8dbcba8480f7e6776f04168ed7caaa5861dc9" Nov 03 21:54:57.122208 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.122174 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf"] Nov 03 21:54:57.122901 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.122878 5041 cpu_manager.go:414] "containerMap: removing container" podUID="191390f3-d9fb-4471-99ee-a569ef09dca5" containerName="revoke-keys" Nov 03 21:54:57.123006 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.122902 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191390f3-d9fb-4471-99ee-a569ef09dca5" containerName="revoke-keys" Nov 03 21:54:57.123096 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.123081 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="191390f3-d9fb-4471-99ee-a569ef09dca5" containerName="revoke-keys" Nov 03 21:54:57.126012 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.125985 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" Nov 03 21:54:57.147542 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.147509 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf"] Nov 03 21:54:57.158944 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.158907 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:54:57.159119 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.158995 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:54:57.261025 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.260990 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052930eb-fb57-45d2-b4aa-db8bb6cf85db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-z9fmf\" (UID: \"052930eb-fb57-45d2-b4aa-db8bb6cf85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" Nov 03 21:54:57.261232 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.261045 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052930eb-fb57-45d2-b4aa-db8bb6cf85db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-z9fmf\" (UID: \"052930eb-fb57-45d2-b4aa-db8bb6cf85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" Nov 03 21:54:57.261232 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.261167 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qv9p\" (UniqueName: \"kubernetes.io/projected/052930eb-fb57-45d2-b4aa-db8bb6cf85db-kube-api-access-6qv9p\") pod \"maas-api-revoke-sub-5a29efb3-z9fmf\" (UID: \"052930eb-fb57-45d2-b4aa-db8bb6cf85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" Nov 03 21:54:57.362519 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.362483 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052930eb-fb57-45d2-b4aa-db8bb6cf85db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-z9fmf\" (UID: \"052930eb-fb57-45d2-b4aa-db8bb6cf85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" Nov 03 21:54:57.362730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.362541 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qv9p\" (UniqueName: \"kubernetes.io/projected/052930eb-fb57-45d2-b4aa-db8bb6cf85db-kube-api-access-6qv9p\") pod \"maas-api-revoke-sub-5a29efb3-z9fmf\" (UID: \"052930eb-fb57-45d2-b4aa-db8bb6cf85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" Nov 03 21:54:57.362730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.362609 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052930eb-fb57-45d2-b4aa-db8bb6cf85db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-z9fmf\" (UID: \"052930eb-fb57-45d2-b4aa-db8bb6cf85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" Nov 03 21:54:57.363337 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.363312 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052930eb-fb57-45d2-b4aa-db8bb6cf85db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-z9fmf\" (UID: \"052930eb-fb57-45d2-b4aa-db8bb6cf85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" Nov 03 21:54:57.364984 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.364961 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052930eb-fb57-45d2-b4aa-db8bb6cf85db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-z9fmf\" (UID: \"052930eb-fb57-45d2-b4aa-db8bb6cf85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" Nov 03 21:54:57.383378 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.383317 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qv9p\" (UniqueName: \"kubernetes.io/projected/052930eb-fb57-45d2-b4aa-db8bb6cf85db-kube-api-access-6qv9p\") pod \"maas-api-revoke-sub-5a29efb3-z9fmf\" (UID: \"052930eb-fb57-45d2-b4aa-db8bb6cf85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" Nov 03 21:54:57.450098 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.450071 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" Nov 03 21:54:57.747129 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:54:57.747091 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052930eb_fb57_45d2_b4aa_db8bb6cf85db.slice/crio-77df565c97f10a898714efbf14a63bddd557f225813204b632d434c440c18226 WatchSource:0}: Error finding container 77df565c97f10a898714efbf14a63bddd557f225813204b632d434c440c18226: Status 404 returned error can't find the container with id 77df565c97f10a898714efbf14a63bddd557f225813204b632d434c440c18226 Nov 03 21:54:57.747626 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:57.747600 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf"] Nov 03 21:54:58.208258 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:58.208217 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" event={"ID":"052930eb-fb57-45d2-b4aa-db8bb6cf85db","Type":"ContainerStarted","Data":"4bf4abc1d36d0860d73eb1e164a051cb05816166268a64a88dfef1c366ba09d7"} Nov 03 21:54:58.208686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:58.208284 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" event={"ID":"052930eb-fb57-45d2-b4aa-db8bb6cf85db","Type":"ContainerStarted","Data":"77df565c97f10a898714efbf14a63bddd557f225813204b632d434c440c18226"} Nov 03 21:54:58.248516 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:58.248456 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" podStartSLOduration=1.248442692 podStartE2EDuration="1.248442692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:54:58.247818275 +0000 UTC m=+1513.755046574" watchObservedRunningTime="2026-11-03 21:54:58.248442692 +0000 UTC m=+1513.755670965" Nov 03 21:54:59.222492 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:59.222449 5041 generic.go:356] "Generic (PLEG): container finished" podID="052930eb-fb57-45d2-b4aa-db8bb6cf85db" containerID="4bf4abc1d36d0860d73eb1e164a051cb05816166268a64a88dfef1c366ba09d7" exitCode=0 Nov 03 21:54:59.223091 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:54:59.222525 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" event={"ID":"052930eb-fb57-45d2-b4aa-db8bb6cf85db","Type":"ContainerDied","Data":"4bf4abc1d36d0860d73eb1e164a051cb05816166268a64a88dfef1c366ba09d7"} Nov 03 21:55:01.248686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:01.238919 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" Nov 03 21:55:01.326963 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:01.326623 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" event={"ID":"052930eb-fb57-45d2-b4aa-db8bb6cf85db","Type":"ContainerDied","Data":"77df565c97f10a898714efbf14a63bddd557f225813204b632d434c440c18226"} Nov 03 21:55:01.326963 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:01.326718 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77df565c97f10a898714efbf14a63bddd557f225813204b632d434c440c18226" Nov 03 21:55:01.326963 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:01.326795 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf" Nov 03 21:55:01.402687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:01.402247 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/052930eb-fb57-45d2-b4aa-db8bb6cf85db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052930eb-fb57-45d2-b4aa-db8bb6cf85db-maas-api-openshift-ca\") pod \"052930eb-fb57-45d2-b4aa-db8bb6cf85db\" (UID: \"052930eb-fb57-45d2-b4aa-db8bb6cf85db\") " Nov 03 21:55:01.402687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:01.402390 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/052930eb-fb57-45d2-b4aa-db8bb6cf85db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052930eb-fb57-45d2-b4aa-db8bb6cf85db-maas-api-xks-ca\") pod \"052930eb-fb57-45d2-b4aa-db8bb6cf85db\" (UID: \"052930eb-fb57-45d2-b4aa-db8bb6cf85db\") " Nov 03 21:55:01.402687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:01.402455 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052930eb-fb57-45d2-b4aa-db8bb6cf85db-kube-api-access-6qv9p\" (UniqueName: \"kubernetes.io/projected/052930eb-fb57-45d2-b4aa-db8bb6cf85db-kube-api-access-6qv9p\") pod \"052930eb-fb57-45d2-b4aa-db8bb6cf85db\" (UID: \"052930eb-fb57-45d2-b4aa-db8bb6cf85db\") " Nov 03 21:55:01.425685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:01.420273 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052930eb-fb57-45d2-b4aa-db8bb6cf85db-kube-api-access-6qv9p" pod "052930eb-fb57-45d2-b4aa-db8bb6cf85db" (UID: "052930eb-fb57-45d2-b4aa-db8bb6cf85db"). InnerVolumeSpecName "kube-api-access-6qv9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:55:01.426021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:01.425988 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052930eb-fb57-45d2-b4aa-db8bb6cf85db-maas-api-openshift-ca" pod "052930eb-fb57-45d2-b4aa-db8bb6cf85db" (UID: "052930eb-fb57-45d2-b4aa-db8bb6cf85db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:55:01.450282 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:01.450241 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/052930eb-fb57-45d2-b4aa-db8bb6cf85db-maas-api-xks-ca" pod "052930eb-fb57-45d2-b4aa-db8bb6cf85db" (UID: "052930eb-fb57-45d2-b4aa-db8bb6cf85db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:55:01.504541 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:01.504421 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052930eb-fb57-45d2-b4aa-db8bb6cf85db-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:55:01.504541 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:01.504475 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052930eb-fb57-45d2-b4aa-db8bb6cf85db-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:55:01.504541 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:01.504494 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qv9p\" (UniqueName: \"kubernetes.io/projected/052930eb-fb57-45d2-b4aa-db8bb6cf85db-kube-api-access-6qv9p\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:55:10.777558 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:55:10.777519 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:55:16.377430 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.377395 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz"] Nov 03 21:55:16.391065 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.391032 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dsvpz"] Nov 03 21:55:16.423321 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.423290 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl"] Nov 03 21:55:16.423826 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.423812 5041 cpu_manager.go:414] "containerMap: removing container" podUID="052930eb-fb57-45d2-b4aa-db8bb6cf85db" containerName="revoke-keys" Nov 03 21:55:16.423887 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.423827 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052930eb-fb57-45d2-b4aa-db8bb6cf85db" containerName="revoke-keys" Nov 03 21:55:16.423923 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.423903 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="052930eb-fb57-45d2-b4aa-db8bb6cf85db" containerName="revoke-keys" Nov 03 21:55:16.426491 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.426466 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" Nov 03 21:55:16.442224 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.442178 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:55:16.442404 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.442380 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:55:16.443446 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.443421 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl"] Nov 03 21:55:16.458468 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.458433 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1760b9ce-efd4-4674-a7a3-6c226efd82c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zztfl\" (UID: \"1760b9ce-efd4-4674-a7a3-6c226efd82c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" Nov 03 21:55:16.458628 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.458502 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1760b9ce-efd4-4674-a7a3-6c226efd82c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zztfl\" (UID: \"1760b9ce-efd4-4674-a7a3-6c226efd82c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" Nov 03 21:55:16.458714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.458671 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t552z\" (UniqueName: \"kubernetes.io/projected/1760b9ce-efd4-4674-a7a3-6c226efd82c1-kube-api-access-t552z\") pod \"maas-api-revoke-sub-3d4156f1-zztfl\" (UID: \"1760b9ce-efd4-4674-a7a3-6c226efd82c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" Nov 03 21:55:16.560186 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.560143 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1760b9ce-efd4-4674-a7a3-6c226efd82c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zztfl\" (UID: \"1760b9ce-efd4-4674-a7a3-6c226efd82c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" Nov 03 21:55:16.560386 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.560203 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1760b9ce-efd4-4674-a7a3-6c226efd82c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zztfl\" (UID: \"1760b9ce-efd4-4674-a7a3-6c226efd82c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" Nov 03 21:55:16.560386 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.560233 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t552z\" (UniqueName: \"kubernetes.io/projected/1760b9ce-efd4-4674-a7a3-6c226efd82c1-kube-api-access-t552z\") pod \"maas-api-revoke-sub-3d4156f1-zztfl\" (UID: \"1760b9ce-efd4-4674-a7a3-6c226efd82c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" Nov 03 21:55:16.560957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.560921 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1760b9ce-efd4-4674-a7a3-6c226efd82c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zztfl\" (UID: \"1760b9ce-efd4-4674-a7a3-6c226efd82c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" Nov 03 21:55:16.562478 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.562448 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1760b9ce-efd4-4674-a7a3-6c226efd82c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zztfl\" (UID: \"1760b9ce-efd4-4674-a7a3-6c226efd82c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" Nov 03 21:55:16.580624 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.580598 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t552z\" (UniqueName: \"kubernetes.io/projected/1760b9ce-efd4-4674-a7a3-6c226efd82c1-kube-api-access-t552z\") pod \"maas-api-revoke-sub-3d4156f1-zztfl\" (UID: \"1760b9ce-efd4-4674-a7a3-6c226efd82c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" Nov 03 21:55:16.621601 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.621572 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f63df7-0aab-44cc-9703-a0613ad73195" path="/var/lib/kubelet/pods/a5f63df7-0aab-44cc-9703-a0613ad73195/volumes" Nov 03 21:55:16.739607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:16.739563 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" Nov 03 21:55:17.034428 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:17.034403 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl"] Nov 03 21:55:17.035448 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:55:17.035423 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1760b9ce_efd4_4674_a7a3_6c226efd82c1.slice/crio-809407d188ce4f9bf6688105fb5a04a322c98159bded756b23ca8fc631f12c9d WatchSource:0}: Error finding container 809407d188ce4f9bf6688105fb5a04a322c98159bded756b23ca8fc631f12c9d: Status 404 returned error can't find the container with id 809407d188ce4f9bf6688105fb5a04a322c98159bded756b23ca8fc631f12c9d Nov 03 21:55:17.435670 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:17.435626 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" event={"ID":"1760b9ce-efd4-4674-a7a3-6c226efd82c1","Type":"ContainerStarted","Data":"eae8172444daeecec585d259d53c71791c1af8ba1679e52c32b81c49987e2686"} Nov 03 21:55:17.436117 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:17.435709 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" event={"ID":"1760b9ce-efd4-4674-a7a3-6c226efd82c1","Type":"ContainerStarted","Data":"809407d188ce4f9bf6688105fb5a04a322c98159bded756b23ca8fc631f12c9d"} Nov 03 21:55:17.476558 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:17.476483 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" podStartSLOduration=1.47647055 podStartE2EDuration="1.47647055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:55:17.475207804 +0000 UTC m=+1532.982436084" watchObservedRunningTime="2026-11-03 21:55:17.47647055 +0000 UTC m=+1532.983698812" Nov 03 21:55:18.443602 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:18.443511 5041 generic.go:356] "Generic (PLEG): container finished" podID="1760b9ce-efd4-4674-a7a3-6c226efd82c1" containerID="eae8172444daeecec585d259d53c71791c1af8ba1679e52c32b81c49987e2686" exitCode=0 Nov 03 21:55:18.444035 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:18.443594 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" event={"ID":"1760b9ce-efd4-4674-a7a3-6c226efd82c1","Type":"ContainerDied","Data":"eae8172444daeecec585d259d53c71791c1af8ba1679e52c32b81c49987e2686"} Nov 03 21:55:19.601233 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:19.601206 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" Nov 03 21:55:19.692657 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:19.692628 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1760b9ce-efd4-4674-a7a3-6c226efd82c1-kube-api-access-t552z\" (UniqueName: \"kubernetes.io/projected/1760b9ce-efd4-4674-a7a3-6c226efd82c1-kube-api-access-t552z\") pod \"1760b9ce-efd4-4674-a7a3-6c226efd82c1\" (UID: \"1760b9ce-efd4-4674-a7a3-6c226efd82c1\") " Nov 03 21:55:19.692845 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:19.692747 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1760b9ce-efd4-4674-a7a3-6c226efd82c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1760b9ce-efd4-4674-a7a3-6c226efd82c1-maas-api-xks-ca\") pod \"1760b9ce-efd4-4674-a7a3-6c226efd82c1\" (UID: \"1760b9ce-efd4-4674-a7a3-6c226efd82c1\") " Nov 03 21:55:19.692845 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:19.692781 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1760b9ce-efd4-4674-a7a3-6c226efd82c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1760b9ce-efd4-4674-a7a3-6c226efd82c1-maas-api-openshift-ca\") pod \"1760b9ce-efd4-4674-a7a3-6c226efd82c1\" (UID: \"1760b9ce-efd4-4674-a7a3-6c226efd82c1\") " Nov 03 21:55:19.693229 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:19.693194 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1760b9ce-efd4-4674-a7a3-6c226efd82c1-maas-api-openshift-ca" pod "1760b9ce-efd4-4674-a7a3-6c226efd82c1" (UID: "1760b9ce-efd4-4674-a7a3-6c226efd82c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:55:19.694886 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:19.694858 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1760b9ce-efd4-4674-a7a3-6c226efd82c1-maas-api-xks-ca" pod "1760b9ce-efd4-4674-a7a3-6c226efd82c1" (UID: "1760b9ce-efd4-4674-a7a3-6c226efd82c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:55:19.694886 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:19.694876 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1760b9ce-efd4-4674-a7a3-6c226efd82c1-kube-api-access-t552z" pod "1760b9ce-efd4-4674-a7a3-6c226efd82c1" (UID: "1760b9ce-efd4-4674-a7a3-6c226efd82c1"). InnerVolumeSpecName "kube-api-access-t552z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:55:19.794158 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:19.794125 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t552z\" (UniqueName: \"kubernetes.io/projected/1760b9ce-efd4-4674-a7a3-6c226efd82c1-kube-api-access-t552z\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:55:19.794158 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:19.794152 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1760b9ce-efd4-4674-a7a3-6c226efd82c1-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:55:19.794158 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:19.794163 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1760b9ce-efd4-4674-a7a3-6c226efd82c1-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:55:20.457238 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:20.457204 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" Nov 03 21:55:20.457238 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:20.457228 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl" event={"ID":"1760b9ce-efd4-4674-a7a3-6c226efd82c1","Type":"ContainerDied","Data":"809407d188ce4f9bf6688105fb5a04a322c98159bded756b23ca8fc631f12c9d"} Nov 03 21:55:20.457424 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:20.457277 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="809407d188ce4f9bf6688105fb5a04a322c98159bded756b23ca8fc631f12c9d" Nov 03 21:55:38.312210 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.312171 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s"] Nov 03 21:55:38.312940 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.312922 5041 cpu_manager.go:414] "containerMap: removing container" podUID="1760b9ce-efd4-4674-a7a3-6c226efd82c1" containerName="revoke-keys" Nov 03 21:55:38.313019 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.312941 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1760b9ce-efd4-4674-a7a3-6c226efd82c1" containerName="revoke-keys" Nov 03 21:55:38.313108 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.313096 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1760b9ce-efd4-4674-a7a3-6c226efd82c1" containerName="revoke-keys" Nov 03 21:55:38.316801 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.316778 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.350334 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.350292 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s"] Nov 03 21:55:38.350850 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.350737 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-9f9b50-openshift-default-dockercfg-9n7c8\"" Nov 03 21:55:38.423540 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.423502 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-workload-certs\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.423704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.423577 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-credential-socket\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.423704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.423649 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-workload-socket\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.423793 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.423724 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istiod-ca-cert\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.423793 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.423777 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-token\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.423858 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.423822 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-data\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.423946 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.423926 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhwm5\" (UniqueName: \"kubernetes.io/projected/26aa9d8a-39a7-41bf-a304-60eeecedd68e-kube-api-access-rhwm5\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.423982 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.423957 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-envoy\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.424045 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.424027 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-podinfo\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.525529 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.525445 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-credential-socket\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.525862 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.525840 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-workload-socket\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.526733 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.526709 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istiod-ca-cert\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.527716 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.527695 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-token\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.527879 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.527862 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-data\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.528020 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.526346 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-credential-socket\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.528020 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.526578 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-workload-socket\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.528212 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.528030 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhwm5\" (UniqueName: \"kubernetes.io/projected/26aa9d8a-39a7-41bf-a304-60eeecedd68e-kube-api-access-rhwm5\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.528212 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.528065 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-envoy\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.528212 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.528117 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-podinfo\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.528212 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.528160 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-workload-certs\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.528456 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.528297 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-data\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.528456 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.527489 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istiod-ca-cert\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.528456 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.528423 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-workload-certs\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.533100 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.533076 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-podinfo\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.536082 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.536057 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-envoy\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.548649 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.548623 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhwm5\" (UniqueName: \"kubernetes.io/projected/26aa9d8a-39a7-41bf-a304-60eeecedd68e-kube-api-access-rhwm5\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.548788 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.548753 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-token\") pod \"e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.636297 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.636222 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:38.950026 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.950000 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s"] Nov 03 21:55:38.951904 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:55:38.951875 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26aa9d8a_39a7_41bf_a304_60eeecedd68e.slice/crio-96cc220dffeef2f15316fb533c5ab19e010c015c79e135fdd2c0339a050eacc6 WatchSource:0}: Error finding container 96cc220dffeef2f15316fb533c5ab19e010c015c79e135fdd2c0339a050eacc6: Status 404 returned error can't find the container with id 96cc220dffeef2f15316fb533c5ab19e010c015c79e135fdd2c0339a050eacc6 Nov 03 21:55:38.954205 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.954167 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:55:38.954335 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.954251 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:55:38.954335 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:38.954295 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:55:39.628599 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:39.628564 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" event={"ID":"26aa9d8a-39a7-41bf-a304-60eeecedd68e","Type":"ContainerStarted","Data":"74a55c722aacd39009f1cfb77f4b756ee9e47a7218bd365bc26ff462eebd7a9d"} Nov 03 21:55:39.629032 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:39.628617 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" event={"ID":"26aa9d8a-39a7-41bf-a304-60eeecedd68e","Type":"ContainerStarted","Data":"96cc220dffeef2f15316fb533c5ab19e010c015c79e135fdd2c0339a050eacc6"} Nov 03 21:55:39.636916 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:39.636891 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:39.702098 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:39.702048 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" podStartSLOduration=1.702035049 podStartE2EDuration="1.702035049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:55:39.686132723 +0000 UTC m=+1555.193361001" watchObservedRunningTime="2026-11-03 21:55:39.702035049 +0000 UTC m=+1555.209263328" Nov 03 21:55:40.641906 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:40.641877 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:40.778290 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:55:40.778250 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:55:41.643634 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:41.643589 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:41.644711 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:41.644649 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:44.037288 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:44.037250 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb"] Nov 03 21:55:44.051535 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:44.051507 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gsqkb"] Nov 03 21:55:44.620853 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:44.620819 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b98d0a76-ce06-45e6-9040-204b7b95a5bb" path="/var/lib/kubelet/pods/b98d0a76-ce06-45e6-9040-204b7b95a5bb/volumes" Nov 03 21:55:48.633705 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:48.633586 5041 scope.go:122] "RemoveContainer" containerID="e2b9363c30ce714ed7e6519b6a11be66ee44292afb63ad19b7c01bd971897622" Nov 03 21:55:48.648380 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:48.648354 5041 scope.go:122] "RemoveContainer" containerID="576e8eea534bb1793735305c1a9c3e1b2427193fa90cbb22dab8f38bc2878920" Nov 03 21:55:57.152989 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:57.152216 5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": read tcp 10.132.0.2:42046->10.132.0.175:15021: read: connection reset by peer" start-of-body= Nov 03 21:55:57.152989 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:57.152299 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" podUID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": read tcp 10.132.0.2:42046->10.132.0.175:15021: read: connection reset by peer" Nov 03 21:55:57.775813 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:57.775775 5041 generic.go:356] "Generic (PLEG): container finished" podID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" containerID="74a55c722aacd39009f1cfb77f4b756ee9e47a7218bd365bc26ff462eebd7a9d" exitCode=0 Nov 03 21:55:57.775990 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:57.775845 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" event={"ID":"26aa9d8a-39a7-41bf-a304-60eeecedd68e","Type":"ContainerDied","Data":"74a55c722aacd39009f1cfb77f4b756ee9e47a7218bd365bc26ff462eebd7a9d"} Nov 03 21:55:57.776281 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:57.776264 5041 scope.go:122] "RemoveContainer" containerID="74a55c722aacd39009f1cfb77f4b756ee9e47a7218bd365bc26ff462eebd7a9d" Nov 03 21:55:57.777454 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:57.777427 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:55:57.777539 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:57.777489 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:55:57.777539 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:57.777522 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:55:58.784281 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:58.784240 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" event={"ID":"26aa9d8a-39a7-41bf-a304-60eeecedd68e","Type":"ContainerStarted","Data":"6b455768e782a5192cb5d82e27e5d17da525bd29d3d4ee32e29e2c565e3a0467"} Nov 03 21:55:59.636977 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:59.636931 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:55:59.637267 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:59.637242 5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:55:59.637351 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:55:59.637291 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" podUID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 21:56:00.637308 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:00.637271 5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:56:00.637828 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:00.637333 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" podUID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 21:56:01.638686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:01.638065 5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:56:01.638686 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:01.638128 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" podUID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 21:56:02.639684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:02.639171 5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:56:02.639684 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:02.639245 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" podUID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 21:56:03.637359 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:03.637316 5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:56:03.637573 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:03.637388 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" podUID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 21:56:04.641108 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:04.641074 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:56:04.641556 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:04.641495 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:56:04.642269 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:04.642246 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:56:08.028715 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:08.028678 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv"] Nov 03 21:56:08.042489 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:08.042455 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t8brv"] Nov 03 21:56:08.629784 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:08.629738 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c14b358c-04e7-42bf-b914-f41d424cad39" path="/var/lib/kubelet/pods/c14b358c-04e7-42bf-b914-f41d424cad39/volumes" Nov 03 21:56:10.777263 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:56:10.777225 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:56:20.027297 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:20.027263 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6"] Nov 03 21:56:20.044297 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:20.044261 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8dn6"] Nov 03 21:56:20.620936 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:20.620901 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d9c173a-a699-4331-86f0-b821d90f79c5" path="/var/lib/kubelet/pods/3d9c173a-a699-4331-86f0-b821d90f79c5/volumes" Nov 03 21:56:31.137550 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:31.137511 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m"] Nov 03 21:56:31.142770 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:31.142747 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m" Nov 03 21:56:31.159699 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:31.159646 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m"] Nov 03 21:56:31.168847 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:31.168819 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:56:31.168847 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:31.168843 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:56:31.240180 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:31.240146 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm8tn\" (UniqueName: \"kubernetes.io/projected/b050a83e-b8e5-429e-a0c5-419f5e20c839-kube-api-access-zm8tn\") pod \"maas-api-revoke-sub-feed85b5-4kr5m\" (UID: \"b050a83e-b8e5-429e-a0c5-419f5e20c839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m" Nov 03 21:56:31.240180 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:31.240185 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b050a83e-b8e5-429e-a0c5-419f5e20c839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feed85b5-4kr5m\" (UID: \"b050a83e-b8e5-429e-a0c5-419f5e20c839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m" Nov 03 21:56:31.240397 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:31.240220 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b050a83e-b8e5-429e-a0c5-419f5e20c839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feed85b5-4kr5m\" (UID: \"b050a83e-b8e5-429e-a0c5-419f5e20c839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m" Nov 03 21:56:31.341400 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:31.341362 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm8tn\" (UniqueName: \"kubernetes.io/projected/b050a83e-b8e5-429e-a0c5-419f5e20c839-kube-api-access-zm8tn\") pod \"maas-api-revoke-sub-feed85b5-4kr5m\" (UID: \"b050a83e-b8e5-429e-a0c5-419f5e20c839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m" Nov 03 21:56:31.341614 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:31.341411 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b050a83e-b8e5-429e-a0c5-419f5e20c839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feed85b5-4kr5m\" (UID: \"b050a83e-b8e5-429e-a0c5-419f5e20c839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m" Nov 03 21:56:31.341614 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:31.341447 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b050a83e-b8e5-429e-a0c5-419f5e20c839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feed85b5-4kr5m\" (UID: \"b050a83e-b8e5-429e-a0c5-419f5e20c839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m" Nov 03 21:56:31.342139 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:31.342118 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b050a83e-b8e5-429e-a0c5-419f5e20c839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feed85b5-4kr5m\" (UID: \"b050a83e-b8e5-429e-a0c5-419f5e20c839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m" Nov 03 21:56:31.343867 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:31.343842 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b050a83e-b8e5-429e-a0c5-419f5e20c839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feed85b5-4kr5m\" (UID: \"b050a83e-b8e5-429e-a0c5-419f5e20c839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m" Nov 03 21:56:31.361913 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:31.361884 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm8tn\" (UniqueName: \"kubernetes.io/projected/b050a83e-b8e5-429e-a0c5-419f5e20c839-kube-api-access-zm8tn\") pod \"maas-api-revoke-sub-feed85b5-4kr5m\" (UID: \"b050a83e-b8e5-429e-a0c5-419f5e20c839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m" Nov 03 21:56:31.456740 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:31.456710 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m" Nov 03 21:56:31.755275 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:31.755247 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m"] Nov 03 21:56:31.756190 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:56:31.756157 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb050a83e_b8e5_429e_a0c5_419f5e20c839.slice/crio-4b00f1c2f86212edb93497ca738f27e7f4d1316fe641c30c92e9c938a6d36085 WatchSource:0}: Error finding container 4b00f1c2f86212edb93497ca738f27e7f4d1316fe641c30c92e9c938a6d36085: Status 404 returned error can't find the container with id 4b00f1c2f86212edb93497ca738f27e7f4d1316fe641c30c92e9c938a6d36085 Nov 03 21:56:32.006696 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:32.006600 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m" event={"ID":"b050a83e-b8e5-429e-a0c5-419f5e20c839","Type":"ContainerStarted","Data":"4b00f1c2f86212edb93497ca738f27e7f4d1316fe641c30c92e9c938a6d36085"} Nov 03 21:56:33.014111 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:33.014074 5041 generic.go:356] "Generic (PLEG): container finished" podID="b050a83e-b8e5-429e-a0c5-419f5e20c839" containerID="e51082780cec778825af4ab20ca3d7a897aaebc1b35d4b36e4b5ffe85622eaaf" exitCode=0 Nov 03 21:56:33.014508 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:33.014147 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m" event={"ID":"b050a83e-b8e5-429e-a0c5-419f5e20c839","Type":"ContainerDied","Data":"e51082780cec778825af4ab20ca3d7a897aaebc1b35d4b36e4b5ffe85622eaaf"} Nov 03 21:56:34.168754 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:34.168725 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m" Nov 03 21:56:34.273649 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:34.273569 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b050a83e-b8e5-429e-a0c5-419f5e20c839-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b050a83e-b8e5-429e-a0c5-419f5e20c839-maas-api-openshift-ca\") pod \"b050a83e-b8e5-429e-a0c5-419f5e20c839\" (UID: \"b050a83e-b8e5-429e-a0c5-419f5e20c839\") " Nov 03 21:56:34.273820 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:34.273712 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b050a83e-b8e5-429e-a0c5-419f5e20c839-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b050a83e-b8e5-429e-a0c5-419f5e20c839-maas-api-xks-ca\") pod \"b050a83e-b8e5-429e-a0c5-419f5e20c839\" (UID: \"b050a83e-b8e5-429e-a0c5-419f5e20c839\") " Nov 03 21:56:34.273889 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:34.273870 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b050a83e-b8e5-429e-a0c5-419f5e20c839-kube-api-access-zm8tn\" (UniqueName: \"kubernetes.io/projected/b050a83e-b8e5-429e-a0c5-419f5e20c839-kube-api-access-zm8tn\") pod \"b050a83e-b8e5-429e-a0c5-419f5e20c839\" (UID: \"b050a83e-b8e5-429e-a0c5-419f5e20c839\") " Nov 03 21:56:34.273957 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:34.273934 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b050a83e-b8e5-429e-a0c5-419f5e20c839-maas-api-openshift-ca" pod "b050a83e-b8e5-429e-a0c5-419f5e20c839" (UID: "b050a83e-b8e5-429e-a0c5-419f5e20c839"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:56:34.274284 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:34.274267 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b050a83e-b8e5-429e-a0c5-419f5e20c839-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:56:34.275968 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:34.275935 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b050a83e-b8e5-429e-a0c5-419f5e20c839-kube-api-access-zm8tn" pod "b050a83e-b8e5-429e-a0c5-419f5e20c839" (UID: "b050a83e-b8e5-429e-a0c5-419f5e20c839"). InnerVolumeSpecName "kube-api-access-zm8tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:56:34.276117 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:34.275963 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b050a83e-b8e5-429e-a0c5-419f5e20c839-maas-api-xks-ca" pod "b050a83e-b8e5-429e-a0c5-419f5e20c839" (UID: "b050a83e-b8e5-429e-a0c5-419f5e20c839"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:56:34.375188 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:34.375153 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b050a83e-b8e5-429e-a0c5-419f5e20c839-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:56:34.375188 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:34.375186 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm8tn\" (UniqueName: \"kubernetes.io/projected/b050a83e-b8e5-429e-a0c5-419f5e20c839-kube-api-access-zm8tn\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:56:35.028991 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:35.028958 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m" Nov 03 21:56:35.029186 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:35.028999 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m" event={"ID":"b050a83e-b8e5-429e-a0c5-419f5e20c839","Type":"ContainerDied","Data":"4b00f1c2f86212edb93497ca738f27e7f4d1316fe641c30c92e9c938a6d36085"} Nov 03 21:56:35.029186 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:35.029056 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b00f1c2f86212edb93497ca738f27e7f4d1316fe641c30c92e9c938a6d36085" Nov 03 21:56:40.778088 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:56:40.778049 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:56:42.464996 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.464958 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg"] Nov 03 21:56:42.465484 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.465469 5041 cpu_manager.go:414] "containerMap: removing container" podUID="b050a83e-b8e5-429e-a0c5-419f5e20c839" containerName="revoke-keys" Nov 03 21:56:42.465529 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.465485 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b050a83e-b8e5-429e-a0c5-419f5e20c839" containerName="revoke-keys" Nov 03 21:56:42.465573 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.465564 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b050a83e-b8e5-429e-a0c5-419f5e20c839" containerName="revoke-keys" Nov 03 21:56:42.468950 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.468931 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" Nov 03 21:56:42.486021 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.485993 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg"] Nov 03 21:56:42.496232 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.496205 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:56:42.496397 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.496233 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:56:42.565064 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.565024 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/941c6f09-099c-4bdc-a408-d85b14abf689-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg\" (UID: \"941c6f09-099c-4bdc-a408-d85b14abf689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" Nov 03 21:56:42.565246 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.565203 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/941c6f09-099c-4bdc-a408-d85b14abf689-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg\" (UID: \"941c6f09-099c-4bdc-a408-d85b14abf689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" Nov 03 21:56:42.565246 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.565234 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfcfg\" (UniqueName: \"kubernetes.io/projected/941c6f09-099c-4bdc-a408-d85b14abf689-kube-api-access-jfcfg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg\" (UID: \"941c6f09-099c-4bdc-a408-d85b14abf689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" Nov 03 21:56:42.666713 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.666592 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/941c6f09-099c-4bdc-a408-d85b14abf689-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg\" (UID: \"941c6f09-099c-4bdc-a408-d85b14abf689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" Nov 03 21:56:42.666713 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.666643 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfcfg\" (UniqueName: \"kubernetes.io/projected/941c6f09-099c-4bdc-a408-d85b14abf689-kube-api-access-jfcfg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg\" (UID: \"941c6f09-099c-4bdc-a408-d85b14abf689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" Nov 03 21:56:42.666948 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.666728 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/941c6f09-099c-4bdc-a408-d85b14abf689-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg\" (UID: \"941c6f09-099c-4bdc-a408-d85b14abf689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" Nov 03 21:56:42.667437 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.667407 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/941c6f09-099c-4bdc-a408-d85b14abf689-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg\" (UID: \"941c6f09-099c-4bdc-a408-d85b14abf689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" Nov 03 21:56:42.669301 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.669275 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/941c6f09-099c-4bdc-a408-d85b14abf689-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg\" (UID: \"941c6f09-099c-4bdc-a408-d85b14abf689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" Nov 03 21:56:42.687067 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.687036 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfcfg\" (UniqueName: \"kubernetes.io/projected/941c6f09-099c-4bdc-a408-d85b14abf689-kube-api-access-jfcfg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg\" (UID: \"941c6f09-099c-4bdc-a408-d85b14abf689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" Nov 03 21:56:42.782771 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:42.782729 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" Nov 03 21:56:43.087857 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:43.087816 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg"] Nov 03 21:56:43.097242 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:56:43.095260 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941c6f09_099c_4bdc_a408_d85b14abf689.slice/crio-bdcf1b596387bb435d0688e3d11fe64abaee714b7e1c2f1d8e5e3558d1e06d28 WatchSource:0}: Error finding container bdcf1b596387bb435d0688e3d11fe64abaee714b7e1c2f1d8e5e3558d1e06d28: Status 404 returned error can't find the container with id bdcf1b596387bb435d0688e3d11fe64abaee714b7e1c2f1d8e5e3558d1e06d28 Nov 03 21:56:44.095159 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:44.095119 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" event={"ID":"941c6f09-099c-4bdc-a408-d85b14abf689","Type":"ContainerStarted","Data":"36affe12e68b805bc49b575568e422a8e88a97ecff6217367a0b71ed5ccd08e4"} Nov 03 21:56:44.095589 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:44.095180 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" event={"ID":"941c6f09-099c-4bdc-a408-d85b14abf689","Type":"ContainerStarted","Data":"bdcf1b596387bb435d0688e3d11fe64abaee714b7e1c2f1d8e5e3558d1e06d28"} Nov 03 21:56:45.103964 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:45.103926 5041 generic.go:356] "Generic (PLEG): container finished" podID="941c6f09-099c-4bdc-a408-d85b14abf689" containerID="36affe12e68b805bc49b575568e422a8e88a97ecff6217367a0b71ed5ccd08e4" exitCode=0 Nov 03 21:56:45.104412 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:45.104006 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" event={"ID":"941c6f09-099c-4bdc-a408-d85b14abf689","Type":"ContainerDied","Data":"36affe12e68b805bc49b575568e422a8e88a97ecff6217367a0b71ed5ccd08e4"} Nov 03 21:56:46.267360 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:46.267333 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" Nov 03 21:56:46.412794 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:46.412673 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/941c6f09-099c-4bdc-a408-d85b14abf689-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/941c6f09-099c-4bdc-a408-d85b14abf689-maas-api-xks-ca\") pod \"941c6f09-099c-4bdc-a408-d85b14abf689\" (UID: \"941c6f09-099c-4bdc-a408-d85b14abf689\") " Nov 03 21:56:46.413004 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:46.412794 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941c6f09-099c-4bdc-a408-d85b14abf689-kube-api-access-jfcfg\" (UniqueName: \"kubernetes.io/projected/941c6f09-099c-4bdc-a408-d85b14abf689-kube-api-access-jfcfg\") pod \"941c6f09-099c-4bdc-a408-d85b14abf689\" (UID: \"941c6f09-099c-4bdc-a408-d85b14abf689\") " Nov 03 21:56:46.413004 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:46.412878 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/941c6f09-099c-4bdc-a408-d85b14abf689-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/941c6f09-099c-4bdc-a408-d85b14abf689-maas-api-openshift-ca\") pod \"941c6f09-099c-4bdc-a408-d85b14abf689\" (UID: \"941c6f09-099c-4bdc-a408-d85b14abf689\") " Nov 03 21:56:46.413270 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:46.413246 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941c6f09-099c-4bdc-a408-d85b14abf689-maas-api-openshift-ca" pod "941c6f09-099c-4bdc-a408-d85b14abf689" (UID: "941c6f09-099c-4bdc-a408-d85b14abf689"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:56:46.415131 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:46.415089 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/941c6f09-099c-4bdc-a408-d85b14abf689-maas-api-xks-ca" pod "941c6f09-099c-4bdc-a408-d85b14abf689" (UID: "941c6f09-099c-4bdc-a408-d85b14abf689"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:56:46.415416 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:46.415388 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941c6f09-099c-4bdc-a408-d85b14abf689-kube-api-access-jfcfg" pod "941c6f09-099c-4bdc-a408-d85b14abf689" (UID: "941c6f09-099c-4bdc-a408-d85b14abf689"). InnerVolumeSpecName "kube-api-access-jfcfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:56:46.514155 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:46.514113 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/941c6f09-099c-4bdc-a408-d85b14abf689-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:56:46.514155 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:46.514154 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfcfg\" (UniqueName: \"kubernetes.io/projected/941c6f09-099c-4bdc-a408-d85b14abf689-kube-api-access-jfcfg\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:56:46.514361 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:46.514168 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/941c6f09-099c-4bdc-a408-d85b14abf689-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:56:47.118225 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:47.118188 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" Nov 03 21:56:47.118225 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:47.118205 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg" event={"ID":"941c6f09-099c-4bdc-a408-d85b14abf689","Type":"ContainerDied","Data":"bdcf1b596387bb435d0688e3d11fe64abaee714b7e1c2f1d8e5e3558d1e06d28"} Nov 03 21:56:47.118442 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:47.118286 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdcf1b596387bb435d0688e3d11fe64abaee714b7e1c2f1d8e5e3558d1e06d28" Nov 03 21:56:48.679685 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:48.679625 5041 scope.go:122] "RemoveContainer" containerID="e51dc825d2eae148d3e39faa8637f787f23208d35d002298be17797ea732411e" Nov 03 21:56:53.032452 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:53.032409 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv"] Nov 03 21:56:53.045960 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:53.045920 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-j4bcv"] Nov 03 21:56:54.161317 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:54.161279 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg"] Nov 03 21:56:54.174947 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:54.174904 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9f9b50-hxfxg"] Nov 03 21:56:54.620815 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:54.620777 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eaa0e47-926c-4621-82ce-d9a022c2cfbd" path="/var/lib/kubelet/pods/7eaa0e47-926c-4621-82ce-d9a022c2cfbd/volumes" Nov 03 21:56:54.621178 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:56:54.621164 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="941c6f09-099c-4bdc-a408-d85b14abf689" path="/var/lib/kubelet/pods/941c6f09-099c-4bdc-a408-d85b14abf689/volumes" Nov 03 21:57:00.720187 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:00.720142 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s"] Nov 03 21:57:00.720607 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:00.720418 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" podUID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" containerName="istio-proxy" containerID="cri-o://6b455768e782a5192cb5d82e27e5d17da525bd29d3d4ee32e29e2c565e3a0467" gracePeriod=30 Nov 03 21:57:01.931978 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:01.931943 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl"] Nov 03 21:57:01.945557 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:01.945530 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zztfl"] Nov 03 21:57:01.983427 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:01.983390 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp"] Nov 03 21:57:01.984138 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:01.984107 5041 cpu_manager.go:414] "containerMap: removing container" podUID="941c6f09-099c-4bdc-a408-d85b14abf689" containerName="revoke-keys" Nov 03 21:57:01.984138 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:01.984131 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941c6f09-099c-4bdc-a408-d85b14abf689" containerName="revoke-keys" Nov 03 21:57:01.984354 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:01.984306 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="941c6f09-099c-4bdc-a408-d85b14abf689" containerName="revoke-keys" Nov 03 21:57:01.989461 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:01.989440 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" Nov 03 21:57:02.005128 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:02.005098 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp"] Nov 03 21:57:02.005408 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:02.005385 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:57:02.005538 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:02.005452 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:57:02.080918 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:02.080865 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c7e1963-bf7d-47ae-acf4-253d651e9133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vsjfp\" (UID: \"8c7e1963-bf7d-47ae-acf4-253d651e9133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" Nov 03 21:57:02.081107 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:02.081025 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c7e1963-bf7d-47ae-acf4-253d651e9133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vsjfp\" (UID: \"8c7e1963-bf7d-47ae-acf4-253d651e9133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" Nov 03 21:57:02.081107 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:02.081057 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff5wr\" (UniqueName: \"kubernetes.io/projected/8c7e1963-bf7d-47ae-acf4-253d651e9133-kube-api-access-ff5wr\") pod \"maas-api-revoke-sub-3d4156f1-vsjfp\" (UID: \"8c7e1963-bf7d-47ae-acf4-253d651e9133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" Nov 03 21:57:02.181568 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:02.181523 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c7e1963-bf7d-47ae-acf4-253d651e9133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vsjfp\" (UID: \"8c7e1963-bf7d-47ae-acf4-253d651e9133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" Nov 03 21:57:02.181778 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:02.181690 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c7e1963-bf7d-47ae-acf4-253d651e9133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vsjfp\" (UID: \"8c7e1963-bf7d-47ae-acf4-253d651e9133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" Nov 03 21:57:02.181778 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:02.181742 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff5wr\" (UniqueName: \"kubernetes.io/projected/8c7e1963-bf7d-47ae-acf4-253d651e9133-kube-api-access-ff5wr\") pod \"maas-api-revoke-sub-3d4156f1-vsjfp\" (UID: \"8c7e1963-bf7d-47ae-acf4-253d651e9133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" Nov 03 21:57:02.182353 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:02.182289 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c7e1963-bf7d-47ae-acf4-253d651e9133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vsjfp\" (UID: \"8c7e1963-bf7d-47ae-acf4-253d651e9133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" Nov 03 21:57:02.184099 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:02.184078 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c7e1963-bf7d-47ae-acf4-253d651e9133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vsjfp\" (UID: \"8c7e1963-bf7d-47ae-acf4-253d651e9133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" Nov 03 21:57:02.201565 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:02.201531 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff5wr\" (UniqueName: \"kubernetes.io/projected/8c7e1963-bf7d-47ae-acf4-253d651e9133-kube-api-access-ff5wr\") pod \"maas-api-revoke-sub-3d4156f1-vsjfp\" (UID: \"8c7e1963-bf7d-47ae-acf4-253d651e9133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" Nov 03 21:57:02.303780 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:02.303745 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" Nov 03 21:57:02.603526 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:02.603497 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp"] Nov 03 21:57:02.604472 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:57:02.604431 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7e1963_bf7d_47ae_acf4_253d651e9133.slice/crio-cdebc1baded51753b387e24ee6b8eb5084f09941770026b0e3f1cbce5f00514d WatchSource:0}: Error finding container cdebc1baded51753b387e24ee6b8eb5084f09941770026b0e3f1cbce5f00514d: Status 404 returned error can't find the container with id cdebc1baded51753b387e24ee6b8eb5084f09941770026b0e3f1cbce5f00514d Nov 03 21:57:02.624959 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:02.624923 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1760b9ce-efd4-4674-a7a3-6c226efd82c1" path="/var/lib/kubelet/pods/1760b9ce-efd4-4674-a7a3-6c226efd82c1/volumes" Nov 03 21:57:03.230508 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:03.230472 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" event={"ID":"8c7e1963-bf7d-47ae-acf4-253d651e9133","Type":"ContainerStarted","Data":"fa2cf4058465f21f71a8687c6c0ad03e4edbbe17697394db75a06d106df77ca8"} Nov 03 21:57:03.230926 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:03.230532 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" event={"ID":"8c7e1963-bf7d-47ae-acf4-253d651e9133","Type":"ContainerStarted","Data":"cdebc1baded51753b387e24ee6b8eb5084f09941770026b0e3f1cbce5f00514d"} Nov 03 21:57:03.270471 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:03.270423 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" podStartSLOduration=2.270406906 podStartE2EDuration="2.270406906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:57:03.270044611 +0000 UTC m=+1638.777272919" watchObservedRunningTime="2026-11-03 21:57:03.270406906 +0000 UTC m=+1638.777635189" Nov 03 21:57:04.025855 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:04.025816 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx"] Nov 03 21:57:04.039727 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:04.039690 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8l5lx"] Nov 03 21:57:04.238470 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:04.238428 5041 generic.go:356] "Generic (PLEG): container finished" podID="8c7e1963-bf7d-47ae-acf4-253d651e9133" containerID="fa2cf4058465f21f71a8687c6c0ad03e4edbbe17697394db75a06d106df77ca8" exitCode=0 Nov 03 21:57:04.238961 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:04.238510 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" event={"ID":"8c7e1963-bf7d-47ae-acf4-253d651e9133","Type":"ContainerDied","Data":"fa2cf4058465f21f71a8687c6c0ad03e4edbbe17697394db75a06d106df77ca8"} Nov 03 21:57:04.622654 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:04.622617 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c98657-51d8-4b44-bf28-19279813860c" path="/var/lib/kubelet/pods/d6c98657-51d8-4b44-bf28-19279813860c/volumes" Nov 03 21:57:04.643888 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:04.643855 5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:57:04.644063 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:04.643904 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" podUID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:57:05.396725 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:05.396693 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" Nov 03 21:57:05.412077 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:05.412043 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c7e1963-bf7d-47ae-acf4-253d651e9133-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c7e1963-bf7d-47ae-acf4-253d651e9133-maas-api-openshift-ca\") pod \"8c7e1963-bf7d-47ae-acf4-253d651e9133\" (UID: \"8c7e1963-bf7d-47ae-acf4-253d651e9133\") " Nov 03 21:57:05.412393 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:05.412124 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c7e1963-bf7d-47ae-acf4-253d651e9133-kube-api-access-ff5wr\" (UniqueName: \"kubernetes.io/projected/8c7e1963-bf7d-47ae-acf4-253d651e9133-kube-api-access-ff5wr\") pod \"8c7e1963-bf7d-47ae-acf4-253d651e9133\" (UID: \"8c7e1963-bf7d-47ae-acf4-253d651e9133\") " Nov 03 21:57:05.412393 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:05.412178 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c7e1963-bf7d-47ae-acf4-253d651e9133-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c7e1963-bf7d-47ae-acf4-253d651e9133-maas-api-xks-ca\") pod \"8c7e1963-bf7d-47ae-acf4-253d651e9133\" (UID: \"8c7e1963-bf7d-47ae-acf4-253d651e9133\") " Nov 03 21:57:05.412562 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:05.412441 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c7e1963-bf7d-47ae-acf4-253d651e9133-maas-api-openshift-ca" pod "8c7e1963-bf7d-47ae-acf4-253d651e9133" (UID: "8c7e1963-bf7d-47ae-acf4-253d651e9133"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:57:05.412625 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:05.412562 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c7e1963-bf7d-47ae-acf4-253d651e9133-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:57:05.414460 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:05.414431 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7e1963-bf7d-47ae-acf4-253d651e9133-kube-api-access-ff5wr" pod "8c7e1963-bf7d-47ae-acf4-253d651e9133" (UID: "8c7e1963-bf7d-47ae-acf4-253d651e9133"). InnerVolumeSpecName "kube-api-access-ff5wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:57:05.414460 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:05.414440 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7e1963-bf7d-47ae-acf4-253d651e9133-maas-api-xks-ca" pod "8c7e1963-bf7d-47ae-acf4-253d651e9133" (UID: "8c7e1963-bf7d-47ae-acf4-253d651e9133"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:57:05.514265 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:05.514168 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff5wr\" (UniqueName: \"kubernetes.io/projected/8c7e1963-bf7d-47ae-acf4-253d651e9133-kube-api-access-ff5wr\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:57:05.514265 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:05.514200 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c7e1963-bf7d-47ae-acf4-253d651e9133-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:57:05.972286 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:05.972256 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:57:06.019316 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.019283 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-credential-socket\") pod \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " Nov 03 21:57:06.019510 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.019326 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-envoy\") pod \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " Nov 03 21:57:06.019510 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.019381 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-token\" (UniqueName: \"kubernetes.io/projected/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-token\") pod \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " Nov 03 21:57:06.019510 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.019406 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-podinfo\") pod \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " Nov 03 21:57:06.019510 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.019454 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26aa9d8a-39a7-41bf-a304-60eeecedd68e-kube-api-access-rhwm5\" (UniqueName: \"kubernetes.io/projected/26aa9d8a-39a7-41bf-a304-60eeecedd68e-kube-api-access-rhwm5\") pod \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " Nov 03 21:57:06.019510 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.019481 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-data\") pod \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " Nov 03 21:57:06.019824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.019541 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-workload-socket\") pod \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " Nov 03 21:57:06.019824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.019615 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-workload-certs\") pod \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " Nov 03 21:57:06.019824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.019648 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istiod-ca-cert\") pod \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\" (UID: \"26aa9d8a-39a7-41bf-a304-60eeecedd68e\") " Nov 03 21:57:06.019824 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.019715 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-credential-socket" pod "26aa9d8a-39a7-41bf-a304-60eeecedd68e" (UID: "26aa9d8a-39a7-41bf-a304-60eeecedd68e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:57:06.020047 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.019938 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-workload-socket" pod "26aa9d8a-39a7-41bf-a304-60eeecedd68e" (UID: "26aa9d8a-39a7-41bf-a304-60eeecedd68e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:57:06.020197 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.020148 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:57:06.020197 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.020175 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:57:06.020326 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.020245 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-data" pod "26aa9d8a-39a7-41bf-a304-60eeecedd68e" (UID: "26aa9d8a-39a7-41bf-a304-60eeecedd68e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:57:06.020326 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.020272 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istiod-ca-cert" pod "26aa9d8a-39a7-41bf-a304-60eeecedd68e" (UID: "26aa9d8a-39a7-41bf-a304-60eeecedd68e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:57:06.020441 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.020328 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-workload-certs" pod "26aa9d8a-39a7-41bf-a304-60eeecedd68e" (UID: "26aa9d8a-39a7-41bf-a304-60eeecedd68e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:57:06.021847 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.021817 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-podinfo" pod "26aa9d8a-39a7-41bf-a304-60eeecedd68e" (UID: "26aa9d8a-39a7-41bf-a304-60eeecedd68e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:57:06.021847 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.021843 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-token" pod "26aa9d8a-39a7-41bf-a304-60eeecedd68e" (UID: "26aa9d8a-39a7-41bf-a304-60eeecedd68e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:57:06.022035 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.022014 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26aa9d8a-39a7-41bf-a304-60eeecedd68e-kube-api-access-rhwm5" pod "26aa9d8a-39a7-41bf-a304-60eeecedd68e" (UID: "26aa9d8a-39a7-41bf-a304-60eeecedd68e"). InnerVolumeSpecName "kube-api-access-rhwm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:57:06.022154 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.022139 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-envoy" pod "26aa9d8a-39a7-41bf-a304-60eeecedd68e" (UID: "26aa9d8a-39a7-41bf-a304-60eeecedd68e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:57:06.121278 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.121191 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:57:06.121278 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.121235 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:57:06.121278 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.121252 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhwm5\" (UniqueName: \"kubernetes.io/projected/26aa9d8a-39a7-41bf-a304-60eeecedd68e-kube-api-access-rhwm5\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:57:06.121278 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.121266 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:57:06.121278 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.121279 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:57:06.121591 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.121292 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:57:06.121591 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.121305 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26aa9d8a-39a7-41bf-a304-60eeecedd68e-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:57:06.253566 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.253533 5041 generic.go:356] "Generic (PLEG): container finished" podID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" containerID="6b455768e782a5192cb5d82e27e5d17da525bd29d3d4ee32e29e2c565e3a0467" exitCode=0 Nov 03 21:57:06.253775 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.253589 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" Nov 03 21:57:06.253775 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.253619 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" event={"ID":"26aa9d8a-39a7-41bf-a304-60eeecedd68e","Type":"ContainerDied","Data":"6b455768e782a5192cb5d82e27e5d17da525bd29d3d4ee32e29e2c565e3a0467"} Nov 03 21:57:06.253775 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.253706 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s" event={"ID":"26aa9d8a-39a7-41bf-a304-60eeecedd68e","Type":"ContainerDied","Data":"96cc220dffeef2f15316fb533c5ab19e010c015c79e135fdd2c0339a050eacc6"} Nov 03 21:57:06.253775 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.253737 5041 scope.go:122] "RemoveContainer" containerID="6b455768e782a5192cb5d82e27e5d17da525bd29d3d4ee32e29e2c565e3a0467" Nov 03 21:57:06.255487 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.255461 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" Nov 03 21:57:06.255601 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.255487 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp" event={"ID":"8c7e1963-bf7d-47ae-acf4-253d651e9133","Type":"ContainerDied","Data":"cdebc1baded51753b387e24ee6b8eb5084f09941770026b0e3f1cbce5f00514d"} Nov 03 21:57:06.255601 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.255528 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdebc1baded51753b387e24ee6b8eb5084f09941770026b0e3f1cbce5f00514d" Nov 03 21:57:06.265565 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.265545 5041 scope.go:122] "RemoveContainer" containerID="74a55c722aacd39009f1cfb77f4b756ee9e47a7218bd365bc26ff462eebd7a9d" Nov 03 21:57:06.276615 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.276594 5041 scope.go:122] "RemoveContainer" containerID="6b455768e782a5192cb5d82e27e5d17da525bd29d3d4ee32e29e2c565e3a0467" Nov 03 21:57:06.277007 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:57:06.276976 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b455768e782a5192cb5d82e27e5d17da525bd29d3d4ee32e29e2c565e3a0467\": container with ID starting with 6b455768e782a5192cb5d82e27e5d17da525bd29d3d4ee32e29e2c565e3a0467 not found: ID does not exist" containerID="6b455768e782a5192cb5d82e27e5d17da525bd29d3d4ee32e29e2c565e3a0467" Nov 03 21:57:06.277104 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.277020 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b455768e782a5192cb5d82e27e5d17da525bd29d3d4ee32e29e2c565e3a0467"} err="failed to get container status \"6b455768e782a5192cb5d82e27e5d17da525bd29d3d4ee32e29e2c565e3a0467\": rpc error: code = NotFound desc = could not find container \"6b455768e782a5192cb5d82e27e5d17da525bd29d3d4ee32e29e2c565e3a0467\": container with ID starting with 6b455768e782a5192cb5d82e27e5d17da525bd29d3d4ee32e29e2c565e3a0467 not found: ID does not exist" Nov 03 21:57:06.277104 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.277048 5041 scope.go:122] "RemoveContainer" containerID="74a55c722aacd39009f1cfb77f4b756ee9e47a7218bd365bc26ff462eebd7a9d" Nov 03 21:57:06.277346 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:57:06.277310 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a55c722aacd39009f1cfb77f4b756ee9e47a7218bd365bc26ff462eebd7a9d\": container with ID starting with 74a55c722aacd39009f1cfb77f4b756ee9e47a7218bd365bc26ff462eebd7a9d not found: ID does not exist" containerID="74a55c722aacd39009f1cfb77f4b756ee9e47a7218bd365bc26ff462eebd7a9d" Nov 03 21:57:06.277421 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.277356 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a55c722aacd39009f1cfb77f4b756ee9e47a7218bd365bc26ff462eebd7a9d"} err="failed to get container status \"74a55c722aacd39009f1cfb77f4b756ee9e47a7218bd365bc26ff462eebd7a9d\": rpc error: code = NotFound desc = could not find container \"74a55c722aacd39009f1cfb77f4b756ee9e47a7218bd365bc26ff462eebd7a9d\": container with ID starting with 74a55c722aacd39009f1cfb77f4b756ee9e47a7218bd365bc26ff462eebd7a9d not found: ID does not exist" Nov 03 21:57:06.310065 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.310038 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s"] Nov 03 21:57:06.326076 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.320176 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9f9b50-openshift-default-5ffb986b45fmt8s"] Nov 03 21:57:06.621484 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:06.621448 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" path="/var/lib/kubelet/pods/26aa9d8a-39a7-41bf-a304-60eeecedd68e/volumes" Nov 03 21:57:10.777710 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:57:10.777652 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:57:26.085406 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.085319 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49"] Nov 03 21:57:26.085992 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.085621 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" podUID="d41cc544-3a25-4973-aed1-dfe03f8f992b" containerName="manager" containerID="cri-o://ea469bb20f1be010b4a5ebea794984372fb0bd5fb93acfb5e83cc8fc4c90a8ae" gracePeriod=10 Nov 03 21:57:26.352131 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.352067 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" Nov 03 21:57:26.426926 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.426888 5041 generic.go:356] "Generic (PLEG): container finished" podID="d41cc544-3a25-4973-aed1-dfe03f8f992b" containerID="ea469bb20f1be010b4a5ebea794984372fb0bd5fb93acfb5e83cc8fc4c90a8ae" exitCode=0 Nov 03 21:57:26.427134 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.426944 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" Nov 03 21:57:26.427134 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.426984 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" event={"ID":"d41cc544-3a25-4973-aed1-dfe03f8f992b","Type":"ContainerDied","Data":"ea469bb20f1be010b4a5ebea794984372fb0bd5fb93acfb5e83cc8fc4c90a8ae"} Nov 03 21:57:26.427134 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.427070 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49" event={"ID":"d41cc544-3a25-4973-aed1-dfe03f8f992b","Type":"ContainerDied","Data":"67a4b72f8042f7a0d21f69c9f9acb808364b474dd31f03962eb88837213de335"} Nov 03 21:57:26.427134 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.427116 5041 scope.go:122] "RemoveContainer" containerID="ea469bb20f1be010b4a5ebea794984372fb0bd5fb93acfb5e83cc8fc4c90a8ae" Nov 03 21:57:26.440445 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.440420 5041 scope.go:122] "RemoveContainer" containerID="ea469bb20f1be010b4a5ebea794984372fb0bd5fb93acfb5e83cc8fc4c90a8ae" Nov 03 21:57:26.440871 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:57:26.440850 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea469bb20f1be010b4a5ebea794984372fb0bd5fb93acfb5e83cc8fc4c90a8ae\": container with ID starting with ea469bb20f1be010b4a5ebea794984372fb0bd5fb93acfb5e83cc8fc4c90a8ae not found: ID does not exist" containerID="ea469bb20f1be010b4a5ebea794984372fb0bd5fb93acfb5e83cc8fc4c90a8ae" Nov 03 21:57:26.440978 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.440878 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea469bb20f1be010b4a5ebea794984372fb0bd5fb93acfb5e83cc8fc4c90a8ae"} err="failed to get container status \"ea469bb20f1be010b4a5ebea794984372fb0bd5fb93acfb5e83cc8fc4c90a8ae\": rpc error: code = NotFound desc = could not find container \"ea469bb20f1be010b4a5ebea794984372fb0bd5fb93acfb5e83cc8fc4c90a8ae\": container with ID starting with ea469bb20f1be010b4a5ebea794984372fb0bd5fb93acfb5e83cc8fc4c90a8ae not found: ID does not exist" Nov 03 21:57:26.474213 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.474176 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41cc544-3a25-4973-aed1-dfe03f8f992b-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d41cc544-3a25-4973-aed1-dfe03f8f992b-extensions-socket-volume\") pod \"d41cc544-3a25-4973-aed1-dfe03f8f992b\" (UID: \"d41cc544-3a25-4973-aed1-dfe03f8f992b\") " Nov 03 21:57:26.474405 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.474388 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d41cc544-3a25-4973-aed1-dfe03f8f992b-kube-api-access-hz5zr\" (UniqueName: \"kubernetes.io/projected/d41cc544-3a25-4973-aed1-dfe03f8f992b-kube-api-access-hz5zr\") pod \"d41cc544-3a25-4973-aed1-dfe03f8f992b\" (UID: \"d41cc544-3a25-4973-aed1-dfe03f8f992b\") " Nov 03 21:57:26.474620 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.474596 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41cc544-3a25-4973-aed1-dfe03f8f992b-extensions-socket-volume" pod "d41cc544-3a25-4973-aed1-dfe03f8f992b" (UID: "d41cc544-3a25-4973-aed1-dfe03f8f992b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:57:26.474755 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.474738 5041 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d41cc544-3a25-4973-aed1-dfe03f8f992b-extensions-socket-volume\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:57:26.476451 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.476432 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41cc544-3a25-4973-aed1-dfe03f8f992b-kube-api-access-hz5zr" pod "d41cc544-3a25-4973-aed1-dfe03f8f992b" (UID: "d41cc544-3a25-4973-aed1-dfe03f8f992b"). InnerVolumeSpecName "kube-api-access-hz5zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:57:26.575356 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.575320 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz5zr\" (UniqueName: \"kubernetes.io/projected/d41cc544-3a25-4973-aed1-dfe03f8f992b-kube-api-access-hz5zr\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:57:26.770143 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.770107 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49"] Nov 03 21:57:26.783732 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:26.783704 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qnw49"] Nov 03 21:57:27.027071 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:27.027030 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq"] Nov 03 21:57:27.029696 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:27.029672 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rz5jq"] Nov 03 21:57:28.624678 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:28.624638 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="003f4ad4-0acd-4bdd-872e-bcbba93cbaad" path="/var/lib/kubelet/pods/003f4ad4-0acd-4bdd-872e-bcbba93cbaad/volumes" Nov 03 21:57:28.625151 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:28.625134 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d41cc544-3a25-4973-aed1-dfe03f8f992b" path="/var/lib/kubelet/pods/d41cc544-3a25-4973-aed1-dfe03f8f992b/volumes" Nov 03 21:57:40.777588 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:57:40.777542 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:57:48.708015 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:48.707981 5041 scope.go:122] "RemoveContainer" containerID="ffafd7776bddd435cb2b53fef61c56328fc95e20882ffb3aea2b2599be8973f4" Nov 03 21:57:48.717704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:48.717657 5041 scope.go:122] "RemoveContainer" containerID="bcace69a0b8a4be1020b8eadda93dd5157f15215930aad778197ee7f73a1b2d9" Nov 03 21:57:49.026398 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:49.026311 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t"] Nov 03 21:57:49.039650 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:49.039615 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9dn9t"] Nov 03 21:57:50.620512 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:57:50.620471 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aded170-bd02-42f0-90fc-0fcf113d8fe0" path="/var/lib/kubelet/pods/2aded170-bd02-42f0-90fc-0fcf113d8fe0/volumes" Nov 03 21:58:10.777238 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:58:10.777196 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:58:12.026780 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:12.026743 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw"] Nov 03 21:58:12.040048 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:12.040021 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6bztw"] Nov 03 21:58:12.621101 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:12.621063 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb0476f3-a816-494a-b1bb-1da86667bf9c" path="/var/lib/kubelet/pods/cb0476f3-a816-494a-b1bb-1da86667bf9c/volumes" Nov 03 21:58:29.214687 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.214636 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6s2gm"] Nov 03 21:58:29.215425 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.215401 5041 cpu_manager.go:414] "containerMap: removing container" podUID="d41cc544-3a25-4973-aed1-dfe03f8f992b" containerName="manager" Nov 03 21:58:29.215533 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.215426 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d41cc544-3a25-4973-aed1-dfe03f8f992b" containerName="manager" Nov 03 21:58:29.215533 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.215459 5041 cpu_manager.go:414] "containerMap: removing container" podUID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" containerName="istio-proxy" Nov 03 21:58:29.215533 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.215466 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" containerName="istio-proxy" Nov 03 21:58:29.215533 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.215489 5041 cpu_manager.go:414] "containerMap: removing container" podUID="8c7e1963-bf7d-47ae-acf4-253d651e9133" containerName="revoke-keys" Nov 03 21:58:29.215533 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.215496 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c7e1963-bf7d-47ae-acf4-253d651e9133" containerName="revoke-keys" Nov 03 21:58:29.215859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.215646 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d41cc544-3a25-4973-aed1-dfe03f8f992b" containerName="manager" Nov 03 21:58:29.215859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.215676 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" containerName="istio-proxy" Nov 03 21:58:29.215859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.215691 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" containerName="istio-proxy" Nov 03 21:58:29.215859 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.215700 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c7e1963-bf7d-47ae-acf4-253d651e9133" containerName="revoke-keys" Nov 03 21:58:29.219228 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.219208 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6s2gm" Nov 03 21:58:29.238900 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.238871 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6s2gm"] Nov 03 21:58:29.249467 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.249437 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kprcl\"" Nov 03 21:58:29.301179 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.301144 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xztn2\" (UniqueName: \"kubernetes.io/projected/c5b685e7-1863-4500-a5f6-1e58b8335e76-kube-api-access-xztn2\") pod \"kuadrant-operator-controller-manager-678d87766f-6s2gm\" (UID: \"c5b685e7-1863-4500-a5f6-1e58b8335e76\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6s2gm" Nov 03 21:58:29.301387 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.301323 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5b685e7-1863-4500-a5f6-1e58b8335e76-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6s2gm\" (UID: \"c5b685e7-1863-4500-a5f6-1e58b8335e76\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6s2gm" Nov 03 21:58:29.402816 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.402773 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5b685e7-1863-4500-a5f6-1e58b8335e76-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6s2gm\" (UID: \"c5b685e7-1863-4500-a5f6-1e58b8335e76\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6s2gm" Nov 03 21:58:29.403030 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.402847 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xztn2\" (UniqueName: \"kubernetes.io/projected/c5b685e7-1863-4500-a5f6-1e58b8335e76-kube-api-access-xztn2\") pod \"kuadrant-operator-controller-manager-678d87766f-6s2gm\" (UID: \"c5b685e7-1863-4500-a5f6-1e58b8335e76\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6s2gm" Nov 03 21:58:29.403256 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.403228 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5b685e7-1863-4500-a5f6-1e58b8335e76-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6s2gm\" (UID: \"c5b685e7-1863-4500-a5f6-1e58b8335e76\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6s2gm" Nov 03 21:58:29.423704 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.423655 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xztn2\" (UniqueName: \"kubernetes.io/projected/c5b685e7-1863-4500-a5f6-1e58b8335e76-kube-api-access-xztn2\") pod \"kuadrant-operator-controller-manager-678d87766f-6s2gm\" (UID: \"c5b685e7-1863-4500-a5f6-1e58b8335e76\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6s2gm" Nov 03 21:58:29.532479 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.532395 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6s2gm" Nov 03 21:58:29.879119 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:29.879089 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6s2gm"] Nov 03 21:58:29.879714 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:58:29.879687 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b685e7_1863_4500_a5f6_1e58b8335e76.slice/crio-d2c0d650ed6051cbd7f313016a8af5c1497a54ea004a109e9bc9d300c4074a7f WatchSource:0}: Error finding container d2c0d650ed6051cbd7f313016a8af5c1497a54ea004a109e9bc9d300c4074a7f: Status 404 returned error can't find the container with id d2c0d650ed6051cbd7f313016a8af5c1497a54ea004a109e9bc9d300c4074a7f Nov 03 21:58:30.831214 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:30.831172 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6s2gm" event={"ID":"c5b685e7-1863-4500-a5f6-1e58b8335e76","Type":"ContainerStarted","Data":"3c0633c2af10709e6a57e1d91f2db1432b65e8a29a3c77f29109604dab37fb22"} Nov 03 21:58:30.831655 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:30.831240 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6s2gm" event={"ID":"c5b685e7-1863-4500-a5f6-1e58b8335e76","Type":"ContainerStarted","Data":"d2c0d650ed6051cbd7f313016a8af5c1497a54ea004a109e9bc9d300c4074a7f"} Nov 03 21:58:30.831655 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:30.831268 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6s2gm" Nov 03 21:58:30.903636 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:30.903579 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6s2gm" podStartSLOduration=1.903565368 podStartE2EDuration="1.903565368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:58:30.888470114 +0000 UTC m=+1726.395698393" watchObservedRunningTime="2026-11-03 21:58:30.903565368 +0000 UTC m=+1726.410793648" Nov 03 21:58:32.035376 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:32.035335 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr"] Nov 03 21:58:32.046346 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:32.046308 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sx9cr"] Nov 03 21:58:32.621110 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:32.621071 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4cc25f7-b147-42ee-8e76-1d124f2cbc8e" path="/var/lib/kubelet/pods/c4cc25f7-b147-42ee-8e76-1d124f2cbc8e/volumes" Nov 03 21:58:40.778204 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:58:40.778161 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:58:41.840374 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:41.840344 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6s2gm" Nov 03 21:58:48.768606 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:48.768572 5041 scope.go:122] "RemoveContainer" containerID="e2cba7a2c76b6595f417f9bcf05cc90e5ac80cdb9496cbc89572ec27ea0b9e41" Nov 03 21:58:48.802939 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:48.802896 5041 scope.go:122] "RemoveContainer" containerID="9d95f578692a5e826601696c61716cacea801aaaa2bdf6433bff4d54116063b3" Nov 03 21:58:48.819871 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:48.819839 5041 scope.go:122] "RemoveContainer" containerID="bc9c6a97d6152dd10a326343a5550c87cbb9a33d862f5ac55e04c669a4722598" Nov 03 21:58:48.875714 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:48.875684 5041 scope.go:122] "RemoveContainer" containerID="4d4ad06c046f010b130ce21672f7c036a33f91ca0b072f0e36b0697d5c1032ad" Nov 03 21:58:51.070182 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.070133 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb"] Nov 03 21:58:51.070681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.070649 5041 cpu_manager.go:414] "containerMap: removing container" podUID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" containerName="istio-proxy" Nov 03 21:58:51.070681 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.070676 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26aa9d8a-39a7-41bf-a304-60eeecedd68e" containerName="istio-proxy" Nov 03 21:58:51.075727 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.075706 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" Nov 03 21:58:51.091154 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.091117 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb"] Nov 03 21:58:51.101266 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.101237 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:58:51.101406 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.101271 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:58:51.137995 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.137963 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv6cj\" (UniqueName: \"kubernetes.io/projected/07fdee8c-dfe3-4ada-9999-c0e073b7187d-kube-api-access-fv6cj\") pod \"maas-api-revoke-sub-d576e68b-p7szb\" (UID: \"07fdee8c-dfe3-4ada-9999-c0e073b7187d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" Nov 03 21:58:51.138171 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.138038 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07fdee8c-dfe3-4ada-9999-c0e073b7187d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-p7szb\" (UID: \"07fdee8c-dfe3-4ada-9999-c0e073b7187d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" Nov 03 21:58:51.138171 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.138085 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07fdee8c-dfe3-4ada-9999-c0e073b7187d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-p7szb\" (UID: \"07fdee8c-dfe3-4ada-9999-c0e073b7187d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" Nov 03 21:58:51.239427 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.239391 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv6cj\" (UniqueName: \"kubernetes.io/projected/07fdee8c-dfe3-4ada-9999-c0e073b7187d-kube-api-access-fv6cj\") pod \"maas-api-revoke-sub-d576e68b-p7szb\" (UID: \"07fdee8c-dfe3-4ada-9999-c0e073b7187d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" Nov 03 21:58:51.239619 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.239469 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07fdee8c-dfe3-4ada-9999-c0e073b7187d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-p7szb\" (UID: \"07fdee8c-dfe3-4ada-9999-c0e073b7187d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" Nov 03 21:58:51.239619 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.239511 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07fdee8c-dfe3-4ada-9999-c0e073b7187d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-p7szb\" (UID: \"07fdee8c-dfe3-4ada-9999-c0e073b7187d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" Nov 03 21:58:51.240297 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.240275 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07fdee8c-dfe3-4ada-9999-c0e073b7187d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-p7szb\" (UID: \"07fdee8c-dfe3-4ada-9999-c0e073b7187d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" Nov 03 21:58:51.241907 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.241880 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07fdee8c-dfe3-4ada-9999-c0e073b7187d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-p7szb\" (UID: \"07fdee8c-dfe3-4ada-9999-c0e073b7187d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" Nov 03 21:58:51.259822 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.259795 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv6cj\" (UniqueName: \"kubernetes.io/projected/07fdee8c-dfe3-4ada-9999-c0e073b7187d-kube-api-access-fv6cj\") pod \"maas-api-revoke-sub-d576e68b-p7szb\" (UID: \"07fdee8c-dfe3-4ada-9999-c0e073b7187d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" Nov 03 21:58:51.390220 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.390131 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" Nov 03 21:58:51.691122 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:51.691084 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb"] Nov 03 21:58:51.691917 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:58:51.691883 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07fdee8c_dfe3_4ada_9999_c0e073b7187d.slice/crio-59fcc9e6d31fdeef73459ec441f73c4d97d9415784b6f41bf7d95aaa8db6b259 WatchSource:0}: Error finding container 59fcc9e6d31fdeef73459ec441f73c4d97d9415784b6f41bf7d95aaa8db6b259: Status 404 returned error can't find the container with id 59fcc9e6d31fdeef73459ec441f73c4d97d9415784b6f41bf7d95aaa8db6b259 Nov 03 21:58:52.287895 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:52.287791 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" event={"ID":"07fdee8c-dfe3-4ada-9999-c0e073b7187d","Type":"ContainerStarted","Data":"ce19e3bd5855eebe8bf5a26672e5f4b43c4d782ac6d3e601af0028c364310b1e"} Nov 03 21:58:52.287895 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:52.287860 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" event={"ID":"07fdee8c-dfe3-4ada-9999-c0e073b7187d","Type":"ContainerStarted","Data":"59fcc9e6d31fdeef73459ec441f73c4d97d9415784b6f41bf7d95aaa8db6b259"} Nov 03 21:58:52.333388 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:52.333326 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" podStartSLOduration=1.3333112200000001 podStartE2EDuration="1.33331122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:58:52.330247284 +0000 UTC m=+1747.837475597" watchObservedRunningTime="2026-11-03 21:58:52.33331122 +0000 UTC m=+1747.840539499" Nov 03 21:58:53.295822 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:53.295790 5041 generic.go:356] "Generic (PLEG): container finished" podID="07fdee8c-dfe3-4ada-9999-c0e073b7187d" containerID="ce19e3bd5855eebe8bf5a26672e5f4b43c4d782ac6d3e601af0028c364310b1e" exitCode=0 Nov 03 21:58:53.296224 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:53.295848 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" event={"ID":"07fdee8c-dfe3-4ada-9999-c0e073b7187d","Type":"ContainerDied","Data":"ce19e3bd5855eebe8bf5a26672e5f4b43c4d782ac6d3e601af0028c364310b1e"} Nov 03 21:58:54.451793 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:54.451765 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" Nov 03 21:58:54.574344 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:54.574251 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07fdee8c-dfe3-4ada-9999-c0e073b7187d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07fdee8c-dfe3-4ada-9999-c0e073b7187d-maas-api-xks-ca\") pod \"07fdee8c-dfe3-4ada-9999-c0e073b7187d\" (UID: \"07fdee8c-dfe3-4ada-9999-c0e073b7187d\") " Nov 03 21:58:54.574344 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:54.574322 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07fdee8c-dfe3-4ada-9999-c0e073b7187d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07fdee8c-dfe3-4ada-9999-c0e073b7187d-maas-api-openshift-ca\") pod \"07fdee8c-dfe3-4ada-9999-c0e073b7187d\" (UID: \"07fdee8c-dfe3-4ada-9999-c0e073b7187d\") " Nov 03 21:58:54.574551 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:54.574499 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07fdee8c-dfe3-4ada-9999-c0e073b7187d-kube-api-access-fv6cj\" (UniqueName: \"kubernetes.io/projected/07fdee8c-dfe3-4ada-9999-c0e073b7187d-kube-api-access-fv6cj\") pod \"07fdee8c-dfe3-4ada-9999-c0e073b7187d\" (UID: \"07fdee8c-dfe3-4ada-9999-c0e073b7187d\") " Nov 03 21:58:54.574729 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:54.574704 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07fdee8c-dfe3-4ada-9999-c0e073b7187d-maas-api-openshift-ca" pod "07fdee8c-dfe3-4ada-9999-c0e073b7187d" (UID: "07fdee8c-dfe3-4ada-9999-c0e073b7187d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:58:54.574913 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:54.574892 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07fdee8c-dfe3-4ada-9999-c0e073b7187d-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:58:54.576434 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:54.576412 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07fdee8c-dfe3-4ada-9999-c0e073b7187d-maas-api-xks-ca" pod "07fdee8c-dfe3-4ada-9999-c0e073b7187d" (UID: "07fdee8c-dfe3-4ada-9999-c0e073b7187d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:58:54.576673 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:54.576633 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fdee8c-dfe3-4ada-9999-c0e073b7187d-kube-api-access-fv6cj" pod "07fdee8c-dfe3-4ada-9999-c0e073b7187d" (UID: "07fdee8c-dfe3-4ada-9999-c0e073b7187d"). InnerVolumeSpecName "kube-api-access-fv6cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:58:54.675543 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:54.675508 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv6cj\" (UniqueName: \"kubernetes.io/projected/07fdee8c-dfe3-4ada-9999-c0e073b7187d-kube-api-access-fv6cj\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:58:54.675543 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:54.675536 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07fdee8c-dfe3-4ada-9999-c0e073b7187d-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:58:55.315634 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:55.315581 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" Nov 03 21:58:55.315634 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:55.315615 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb" event={"ID":"07fdee8c-dfe3-4ada-9999-c0e073b7187d","Type":"ContainerDied","Data":"59fcc9e6d31fdeef73459ec441f73c4d97d9415784b6f41bf7d95aaa8db6b259"} Nov 03 21:58:55.315886 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:58:55.315766 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59fcc9e6d31fdeef73459ec441f73c4d97d9415784b6f41bf7d95aaa8db6b259" Nov 03 21:59:10.777311 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:59:10.777261 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:59:12.295058 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.294982 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq"] Nov 03 21:59:12.295769 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.295751 5041 cpu_manager.go:414] "containerMap: removing container" podUID="07fdee8c-dfe3-4ada-9999-c0e073b7187d" containerName="revoke-keys" Nov 03 21:59:12.295823 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.295770 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07fdee8c-dfe3-4ada-9999-c0e073b7187d" containerName="revoke-keys" Nov 03 21:59:12.305515 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.296344 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="07fdee8c-dfe3-4ada-9999-c0e073b7187d" containerName="revoke-keys" Nov 03 21:59:12.305515 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.301226 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" Nov 03 21:59:12.314914 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.314885 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq"] Nov 03 21:59:12.321959 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.321934 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db24a304-683e-4536-863c-6d13c244fde3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zxcmq\" (UID: \"db24a304-683e-4536-863c-6d13c244fde3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" Nov 03 21:59:12.322110 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.322047 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db24a304-683e-4536-863c-6d13c244fde3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zxcmq\" (UID: \"db24a304-683e-4536-863c-6d13c244fde3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" Nov 03 21:59:12.322188 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.322117 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl2kr\" (UniqueName: \"kubernetes.io/projected/db24a304-683e-4536-863c-6d13c244fde3-kube-api-access-dl2kr\") pod \"maas-api-revoke-sub-d2bc93bc-zxcmq\" (UID: \"db24a304-683e-4536-863c-6d13c244fde3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" Nov 03 21:59:12.324252 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.324222 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:59:12.324380 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.324271 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 21:59:12.422858 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.422822 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db24a304-683e-4536-863c-6d13c244fde3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zxcmq\" (UID: \"db24a304-683e-4536-863c-6d13c244fde3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" Nov 03 21:59:12.423049 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.422880 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2kr\" (UniqueName: \"kubernetes.io/projected/db24a304-683e-4536-863c-6d13c244fde3-kube-api-access-dl2kr\") pod \"maas-api-revoke-sub-d2bc93bc-zxcmq\" (UID: \"db24a304-683e-4536-863c-6d13c244fde3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" Nov 03 21:59:12.423049 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.422923 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db24a304-683e-4536-863c-6d13c244fde3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zxcmq\" (UID: \"db24a304-683e-4536-863c-6d13c244fde3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" Nov 03 21:59:12.423566 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.423540 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db24a304-683e-4536-863c-6d13c244fde3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zxcmq\" (UID: \"db24a304-683e-4536-863c-6d13c244fde3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" Nov 03 21:59:12.425181 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.425152 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db24a304-683e-4536-863c-6d13c244fde3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zxcmq\" (UID: \"db24a304-683e-4536-863c-6d13c244fde3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" Nov 03 21:59:12.442723 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.442702 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl2kr\" (UniqueName: \"kubernetes.io/projected/db24a304-683e-4536-863c-6d13c244fde3-kube-api-access-dl2kr\") pod \"maas-api-revoke-sub-d2bc93bc-zxcmq\" (UID: \"db24a304-683e-4536-863c-6d13c244fde3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" Nov 03 21:59:12.620984 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.620906 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" Nov 03 21:59:12.914904 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:12.914875 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq"] Nov 03 21:59:12.916618 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:59:12.916588 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb24a304_683e_4536_863c_6d13c244fde3.slice/crio-7f978ed4909bed2f4a57ae62c2202e1b8f0f97b57200a8418cd94666d981ca83 WatchSource:0}: Error finding container 7f978ed4909bed2f4a57ae62c2202e1b8f0f97b57200a8418cd94666d981ca83: Status 404 returned error can't find the container with id 7f978ed4909bed2f4a57ae62c2202e1b8f0f97b57200a8418cd94666d981ca83 Nov 03 21:59:13.458167 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:13.458132 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" event={"ID":"db24a304-683e-4536-863c-6d13c244fde3","Type":"ContainerStarted","Data":"6a2a20c9ab1caa13a94624a9e2945bdaf3d19e55a91922d6d6763356a9645142"} Nov 03 21:59:13.458549 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:13.458196 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" event={"ID":"db24a304-683e-4536-863c-6d13c244fde3","Type":"ContainerStarted","Data":"7f978ed4909bed2f4a57ae62c2202e1b8f0f97b57200a8418cd94666d981ca83"} Nov 03 21:59:13.500860 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:13.500764 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" podStartSLOduration=1.500751051 podStartE2EDuration="1.500751051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:13.497857098 +0000 UTC m=+1769.005085400" watchObservedRunningTime="2026-11-03 21:59:13.500751051 +0000 UTC m=+1769.007979330" Nov 03 21:59:14.466173 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:14.466140 5041 generic.go:356] "Generic (PLEG): container finished" podID="db24a304-683e-4536-863c-6d13c244fde3" containerID="6a2a20c9ab1caa13a94624a9e2945bdaf3d19e55a91922d6d6763356a9645142" exitCode=0 Nov 03 21:59:14.466577 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:14.466231 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" event={"ID":"db24a304-683e-4536-863c-6d13c244fde3","Type":"ContainerDied","Data":"6a2a20c9ab1caa13a94624a9e2945bdaf3d19e55a91922d6d6763356a9645142"} Nov 03 21:59:15.615375 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:15.615342 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" Nov 03 21:59:15.654326 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:15.654297 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db24a304-683e-4536-863c-6d13c244fde3-kube-api-access-dl2kr\" (UniqueName: \"kubernetes.io/projected/db24a304-683e-4536-863c-6d13c244fde3-kube-api-access-dl2kr\") pod \"db24a304-683e-4536-863c-6d13c244fde3\" (UID: \"db24a304-683e-4536-863c-6d13c244fde3\") " Nov 03 21:59:15.654483 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:15.654339 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db24a304-683e-4536-863c-6d13c244fde3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db24a304-683e-4536-863c-6d13c244fde3-maas-api-openshift-ca\") pod \"db24a304-683e-4536-863c-6d13c244fde3\" (UID: \"db24a304-683e-4536-863c-6d13c244fde3\") " Nov 03 21:59:15.654483 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:15.654418 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db24a304-683e-4536-863c-6d13c244fde3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db24a304-683e-4536-863c-6d13c244fde3-maas-api-xks-ca\") pod \"db24a304-683e-4536-863c-6d13c244fde3\" (UID: \"db24a304-683e-4536-863c-6d13c244fde3\") " Nov 03 21:59:15.656234 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:15.656190 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db24a304-683e-4536-863c-6d13c244fde3-maas-api-openshift-ca" pod "db24a304-683e-4536-863c-6d13c244fde3" (UID: "db24a304-683e-4536-863c-6d13c244fde3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:59:15.657025 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:15.656997 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db24a304-683e-4536-863c-6d13c244fde3-kube-api-access-dl2kr" pod "db24a304-683e-4536-863c-6d13c244fde3" (UID: "db24a304-683e-4536-863c-6d13c244fde3"). InnerVolumeSpecName "kube-api-access-dl2kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:59:15.657251 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:15.657227 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db24a304-683e-4536-863c-6d13c244fde3-maas-api-xks-ca" pod "db24a304-683e-4536-863c-6d13c244fde3" (UID: "db24a304-683e-4536-863c-6d13c244fde3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:59:15.756164 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:15.756097 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db24a304-683e-4536-863c-6d13c244fde3-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:59:15.756164 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:15.756124 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl2kr\" (UniqueName: \"kubernetes.io/projected/db24a304-683e-4536-863c-6d13c244fde3-kube-api-access-dl2kr\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:59:15.756164 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:15.756134 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db24a304-683e-4536-863c-6d13c244fde3-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 21:59:16.482807 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:16.482771 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" event={"ID":"db24a304-683e-4536-863c-6d13c244fde3","Type":"ContainerDied","Data":"7f978ed4909bed2f4a57ae62c2202e1b8f0f97b57200a8418cd94666d981ca83"} Nov 03 21:59:16.482807 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:16.482806 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq" Nov 03 21:59:16.483038 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:16.482846 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f978ed4909bed2f4a57ae62c2202e1b8f0f97b57200a8418cd94666d981ca83" Nov 03 21:59:24.419844 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.419810 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j"] Nov 03 21:59:24.420558 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.420537 5041 cpu_manager.go:414] "containerMap: removing container" podUID="db24a304-683e-4536-863c-6d13c244fde3" containerName="revoke-keys" Nov 03 21:59:24.420558 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.420559 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db24a304-683e-4536-863c-6d13c244fde3" containerName="revoke-keys" Nov 03 21:59:24.420753 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.420693 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="db24a304-683e-4536-863c-6d13c244fde3" containerName="revoke-keys" Nov 03 21:59:24.424238 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.424219 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.459747 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.459711 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-3503c0-openshift-default-dockercfg-8lf2s\"" Nov 03 21:59:24.464868 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.464832 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j"] Nov 03 21:59:24.546762 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.546718 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-workload-certs\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.546966 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.546869 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-envoy\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.546966 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.546911 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-podinfo\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.546966 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.546946 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b07d66-ac37-40a9-8a39-7177f2350915-istiod-ca-cert\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.547150 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.546996 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-credential-socket\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.547150 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.547024 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-data\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.547150 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.547080 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-token\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.547150 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.547111 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-workload-socket\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.547470 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.547147 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6f2f\" (UniqueName: \"kubernetes.io/projected/b2b07d66-ac37-40a9-8a39-7177f2350915-kube-api-access-m6f2f\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.647953 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.647915 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-token\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.648243 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.647991 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-workload-socket\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.648396 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.648358 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6f2f\" (UniqueName: \"kubernetes.io/projected/b2b07d66-ac37-40a9-8a39-7177f2350915-kube-api-access-m6f2f\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.648584 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.648418 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-workload-certs\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.648584 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.648555 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-envoy\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.648799 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.648597 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-podinfo\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.648799 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.648632 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b07d66-ac37-40a9-8a39-7177f2350915-istiod-ca-cert\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.648799 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.648715 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-credential-socket\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.648799 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.648740 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-data\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.649028 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.648839 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-workload-socket\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.649028 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.648896 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-workload-certs\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.649140 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.649048 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-data\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.649364 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.649337 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-credential-socket\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.649830 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.649794 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b07d66-ac37-40a9-8a39-7177f2350915-istiod-ca-cert\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.652079 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.652046 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-envoy\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.653923 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.653877 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-podinfo\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.670305 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.670227 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-token\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.670994 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.670971 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6f2f\" (UniqueName: \"kubernetes.io/projected/b2b07d66-ac37-40a9-8a39-7177f2350915-kube-api-access-m6f2f\") pod \"e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:24.740622 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:24.740585 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:25.052341 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:25.052312 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j"] Nov 03 21:59:25.053898 ip-10-0-131-253 kubenswrapper[5041]: W1103 21:59:25.053857 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b07d66_ac37_40a9_8a39_7177f2350915.slice/crio-5f347357c988874bc1d2751b4dc6c9df0bef2b3a194e271d577cda88997d0093 WatchSource:0}: Error finding container 5f347357c988874bc1d2751b4dc6c9df0bef2b3a194e271d577cda88997d0093: Status 404 returned error can't find the container with id 5f347357c988874bc1d2751b4dc6c9df0bef2b3a194e271d577cda88997d0093 Nov 03 21:59:25.056334 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:25.056293 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:59:25.056420 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:25.056383 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:59:25.056457 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:25.056430 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:59:25.548341 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:25.548295 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" event={"ID":"b2b07d66-ac37-40a9-8a39-7177f2350915","Type":"ContainerStarted","Data":"9545b667731c82179832530295c47d95ed02d2fc23efb871d44e88365dab933b"} Nov 03 21:59:25.548836 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:25.548408 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" event={"ID":"b2b07d66-ac37-40a9-8a39-7177f2350915","Type":"ContainerStarted","Data":"5f347357c988874bc1d2751b4dc6c9df0bef2b3a194e271d577cda88997d0093"} Nov 03 21:59:25.593238 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:25.593189 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" podStartSLOduration=1.593175781 podStartE2EDuration="1.593175781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:25.591805518 +0000 UTC m=+1781.099033800" watchObservedRunningTime="2026-11-03 21:59:25.593175781 +0000 UTC m=+1781.100404060" Nov 03 21:59:25.741689 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:25.741624 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:26.745889 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:26.745858 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:27.560934 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:27.560907 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:27.562018 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:27.561998 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:40.781163 ip-10-0-131-253 kubenswrapper[5041]: E1103 21:59:40.781123 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 21:59:42.963952 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:42.963428 5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": read tcp 10.132.0.2:47426->10.132.0.182:15021: read: connection reset by peer" start-of-body= Nov 03 21:59:42.963952 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:42.963536 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" podUID="b2b07d66-ac37-40a9-8a39-7177f2350915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": read tcp 10.132.0.2:47426->10.132.0.182:15021: read: connection reset by peer" Nov 03 21:59:43.691381 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:43.691347 5041 generic.go:356] "Generic (PLEG): container finished" podID="b2b07d66-ac37-40a9-8a39-7177f2350915" containerID="9545b667731c82179832530295c47d95ed02d2fc23efb871d44e88365dab933b" exitCode=0 Nov 03 21:59:43.691575 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:43.691425 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" event={"ID":"b2b07d66-ac37-40a9-8a39-7177f2350915","Type":"ContainerDied","Data":"9545b667731c82179832530295c47d95ed02d2fc23efb871d44e88365dab933b"} Nov 03 21:59:43.691855 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:43.691837 5041 scope.go:122] "RemoveContainer" containerID="9545b667731c82179832530295c47d95ed02d2fc23efb871d44e88365dab933b" Nov 03 21:59:43.693120 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:43.693095 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:59:43.693199 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:43.693163 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:59:43.693238 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:43.693198 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:59:44.699761 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:44.699725 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" event={"ID":"b2b07d66-ac37-40a9-8a39-7177f2350915","Type":"ContainerStarted","Data":"cd6a814acbdd47fdc1b712446074b916fce0df4bf9614f93f850cda16359afb8"} Nov 03 21:59:44.740897 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:44.740865 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:44.741110 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:44.741078 5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 21:59:44.741183 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:44.741130 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" podUID="b2b07d66-ac37-40a9-8a39-7177f2350915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 21:59:45.741730 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:45.741692 5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 21:59:45.742207 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:45.741773 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" podUID="b2b07d66-ac37-40a9-8a39-7177f2350915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 21:59:45.767688 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:45.767644 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 21:59:45.770410 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:45.770386 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 21:59:46.741261 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:46.741222 5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 21:59:46.741447 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:46.741302 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" podUID="b2b07d66-ac37-40a9-8a39-7177f2350915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 21:59:47.741259 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:47.741222 5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 21:59:47.741717 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:47.741279 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" podUID="b2b07d66-ac37-40a9-8a39-7177f2350915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 21:59:48.741466 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:48.741429 5041 patch_prober.go:29] interesting pod/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 21:59:48.741956 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:48.741497 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" podUID="b2b07d66-ac37-40a9-8a39-7177f2350915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 21:59:49.354739 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:49.354705 5041 scope.go:122] "RemoveContainer" containerID="deb5ab0a57ce3b3f0c3c9ed504cf9e577c8da79a164b3f0c6a9a89e66a1b4c72" Nov 03 21:59:49.364989 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:49.364962 5041 scope.go:122] "RemoveContainer" containerID="cecc0efc81c2887b9501d2894372ad82aebba46ef45b037437b2fae271a35ede" Nov 03 21:59:49.744679 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:49.744633 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:49.745166 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:49.745031 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:49.745678 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:49.745642 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 21:59:51.025754 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:51.025715 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6"] Nov 03 21:59:51.039389 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:51.039360 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-wdbm6"] Nov 03 21:59:52.620753 ip-10-0-131-253 kubenswrapper[5041]: I1103 21:59:52.620721 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191390f3-d9fb-4471-99ee-a569ef09dca5" path="/var/lib/kubelet/pods/191390f3-d9fb-4471-99ee-a569ef09dca5/volumes" Nov 03 22:00:00.138796 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:00.138761 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v"] Nov 03 22:00:00.143070 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:00.143051 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" Nov 03 22:00:00.158761 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:00.158736 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v"] Nov 03 22:00:00.168998 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:00.168977 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 22:00:00.274215 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:00.274178 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v\" (UID: \"6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" Nov 03 22:00:00.274215 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:00.274220 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hflnb\" (UniqueName: \"kubernetes.io/projected/6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6-kube-api-access-hflnb\") pod \"maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v\" (UID: \"6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" Nov 03 22:00:00.375530 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:00.375491 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v\" (UID: \"6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" Nov 03 22:00:00.375766 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:00.375539 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hflnb\" (UniqueName: \"kubernetes.io/projected/6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6-kube-api-access-hflnb\") pod \"maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v\" (UID: \"6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" Nov 03 22:00:00.376256 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:00.376231 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v\" (UID: \"6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" Nov 03 22:00:00.395968 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:00.395903 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hflnb\" (UniqueName: \"kubernetes.io/projected/6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6-kube-api-access-hflnb\") pod \"maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v\" (UID: \"6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" Nov 03 22:00:00.458435 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:00.458405 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" Nov 03 22:00:00.757573 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:00.757545 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v"] Nov 03 22:00:00.759209 ip-10-0-131-253 kubenswrapper[5041]: W1103 22:00:00.759179 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5b9a71_89d3_4ea7_97dc_5d3cdf857ed6.slice/crio-3e726194ca41617285ffea07eb21129a91c2f1ff25445e04a1b051fb4a08d92a WatchSource:0}: Error finding container 3e726194ca41617285ffea07eb21129a91c2f1ff25445e04a1b051fb4a08d92a: Status 404 returned error can't find the container with id 3e726194ca41617285ffea07eb21129a91c2f1ff25445e04a1b051fb4a08d92a Nov 03 22:00:00.837072 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:00.837027 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" event={"ID":"6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6","Type":"ContainerStarted","Data":"3e726194ca41617285ffea07eb21129a91c2f1ff25445e04a1b051fb4a08d92a"} Nov 03 22:00:01.845838 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:01.845797 5041 generic.go:356] "Generic (PLEG): container finished" podID="6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" containerID="c30b6250943858406e0f630f1b282a0d45c3bc67c80bc2da3e11362abf9d731b" exitCode=60 Nov 03 22:00:01.846264 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:01.845882 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" event={"ID":"6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6","Type":"ContainerDied","Data":"c30b6250943858406e0f630f1b282a0d45c3bc67c80bc2da3e11362abf9d731b"} Nov 03 22:00:01.846264 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:01.846109 5041 scope.go:122] "RemoveContainer" containerID="c30b6250943858406e0f630f1b282a0d45c3bc67c80bc2da3e11362abf9d731b" Nov 03 22:00:02.033176 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:02.033144 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf"] Nov 03 22:00:02.036930 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:02.036904 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z9fmf"] Nov 03 22:00:02.621153 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:02.621125 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="052930eb-fb57-45d2-b4aa-db8bb6cf85db" path="/var/lib/kubelet/pods/052930eb-fb57-45d2-b4aa-db8bb6cf85db/volumes" Nov 03 22:00:02.854474 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:02.854382 5041 generic.go:356] "Generic (PLEG): container finished" podID="6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" containerID="c219bc622ca6c654f467d944f3dab8cb083e3daa67fc945477c4679f9dd5ef50" exitCode=60 Nov 03 22:00:02.854474 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:02.854438 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" event={"ID":"6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6","Type":"ContainerDied","Data":"c219bc622ca6c654f467d944f3dab8cb083e3daa67fc945477c4679f9dd5ef50"} Nov 03 22:00:02.854984 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:02.854553 5041 scope.go:122] "RemoveContainer" containerID="c30b6250943858406e0f630f1b282a0d45c3bc67c80bc2da3e11362abf9d731b" Nov 03 22:00:02.854984 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:02.854882 5041 scope.go:122] "RemoveContainer" containerID="c219bc622ca6c654f467d944f3dab8cb083e3daa67fc945477c4679f9dd5ef50" Nov 03 22:00:02.855200 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:00:02.855175 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v_odh-ai-gateway-infra(6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" podUID="6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" Nov 03 22:00:03.862610 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:03.862583 5041 scope.go:122] "RemoveContainer" containerID="c219bc622ca6c654f467d944f3dab8cb083e3daa67fc945477c4679f9dd5ef50" Nov 03 22:00:03.863099 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:00:03.862875 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v_odh-ai-gateway-infra(6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" podUID="6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" Nov 03 22:00:10.779870 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:00:10.779827 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:00:15.614209 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:15.614177 5041 scope.go:122] "RemoveContainer" containerID="c219bc622ca6c654f467d944f3dab8cb083e3daa67fc945477c4679f9dd5ef50" Nov 03 22:00:15.941622 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:15.941593 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" event={"ID":"6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6","Type":"ContainerStarted","Data":"c12173604e6d35a1c0e782b288fd6b35e1aff43f0ced73a3147b849deacb8257"} Nov 03 22:00:15.985033 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:15.984979 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" podStartSLOduration=15.984965052 podStartE2EDuration="15.984965052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:15.982289525 +0000 UTC m=+1831.489517815" watchObservedRunningTime="2026-11-03 22:00:15.984965052 +0000 UTC m=+1831.492193331" Nov 03 22:00:16.950436 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:16.950342 5041 generic.go:356] "Generic (PLEG): container finished" podID="6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" containerID="c12173604e6d35a1c0e782b288fd6b35e1aff43f0ced73a3147b849deacb8257" exitCode=60 Nov 03 22:00:16.950436 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:16.950411 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" event={"ID":"6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6","Type":"ContainerDied","Data":"c12173604e6d35a1c0e782b288fd6b35e1aff43f0ced73a3147b849deacb8257"} Nov 03 22:00:16.950972 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:16.950470 5041 scope.go:122] "RemoveContainer" containerID="c219bc622ca6c654f467d944f3dab8cb083e3daa67fc945477c4679f9dd5ef50" Nov 03 22:00:16.950972 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:16.950836 5041 scope.go:122] "RemoveContainer" containerID="c12173604e6d35a1c0e782b288fd6b35e1aff43f0ced73a3147b849deacb8257" Nov 03 22:00:16.951162 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:00:16.951131 5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v_odh-ai-gateway-infra(6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" podUID="6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" Nov 03 22:00:16.993055 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:16.993019 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v"] Nov 03 22:00:18.121159 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:18.121124 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" Nov 03 22:00:18.265725 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:18.265635 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6-maas-api-service-ca\") pod \"6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6\" (UID: \"6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6\") " Nov 03 22:00:18.265873 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:18.265801 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6-kube-api-access-hflnb\" (UniqueName: \"kubernetes.io/projected/6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6-kube-api-access-hflnb\") pod \"6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6\" (UID: \"6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6\") " Nov 03 22:00:18.266043 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:18.266019 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6-maas-api-service-ca" pod "6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" (UID: "6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:18.266186 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:18.266167 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6-maas-api-service-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:00:18.267953 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:18.267927 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6-kube-api-access-hflnb" pod "6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" (UID: "6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6"). InnerVolumeSpecName "kube-api-access-hflnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:18.366942 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:18.366909 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hflnb\" (UniqueName: \"kubernetes.io/projected/6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6-kube-api-access-hflnb\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:00:18.968123 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:18.968097 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" Nov 03 22:00:18.968330 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:18.968129 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v" event={"ID":"6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6","Type":"ContainerDied","Data":"3e726194ca41617285ffea07eb21129a91c2f1ff25445e04a1b051fb4a08d92a"} Nov 03 22:00:18.968330 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:18.968186 5041 scope.go:122] "RemoveContainer" containerID="c12173604e6d35a1c0e782b288fd6b35e1aff43f0ced73a3147b849deacb8257" Nov 03 22:00:19.022567 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:19.022540 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v"] Nov 03 22:00:19.027815 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:19.027791 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-3503c0-2989572wz25v"] Nov 03 22:00:20.621450 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:20.621416 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" path="/var/lib/kubelet/pods/6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6/volumes" Nov 03 22:00:25.064627 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.064593 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn"] Nov 03 22:00:25.065132 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.065118 5041 cpu_manager.go:414] "containerMap: removing container" podUID="6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" containerName="cleanup" Nov 03 22:00:25.065183 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.065134 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" containerName="cleanup" Nov 03 22:00:25.065183 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.065159 5041 cpu_manager.go:414] "containerMap: removing container" podUID="6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" containerName="cleanup" Nov 03 22:00:25.065183 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.065165 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" containerName="cleanup" Nov 03 22:00:25.065280 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.065242 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" containerName="cleanup" Nov 03 22:00:25.065280 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.065257 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" containerName="cleanup" Nov 03 22:00:25.070080 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.070056 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn" Nov 03 22:00:25.086886 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.086851 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn"] Nov 03 22:00:25.096539 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.096500 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:00:25.096749 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.096562 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 22:00:25.139888 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.139420 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhf5h\" (UniqueName: \"kubernetes.io/projected/1082de9c-a6d2-4e01-bc68-385d5d019ce7-kube-api-access-mhf5h\") pod \"maas-api-revoke-sub-5ecc6d17-8n5wn\" (UID: \"1082de9c-a6d2-4e01-bc68-385d5d019ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn" Nov 03 22:00:25.139888 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.139688 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1082de9c-a6d2-4e01-bc68-385d5d019ce7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ecc6d17-8n5wn\" (UID: \"1082de9c-a6d2-4e01-bc68-385d5d019ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn" Nov 03 22:00:25.139888 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.139741 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1082de9c-a6d2-4e01-bc68-385d5d019ce7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ecc6d17-8n5wn\" (UID: \"1082de9c-a6d2-4e01-bc68-385d5d019ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn" Nov 03 22:00:25.241778 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.241739 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1082de9c-a6d2-4e01-bc68-385d5d019ce7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ecc6d17-8n5wn\" (UID: \"1082de9c-a6d2-4e01-bc68-385d5d019ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn" Nov 03 22:00:25.241994 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.241839 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhf5h\" (UniqueName: \"kubernetes.io/projected/1082de9c-a6d2-4e01-bc68-385d5d019ce7-kube-api-access-mhf5h\") pod \"maas-api-revoke-sub-5ecc6d17-8n5wn\" (UID: \"1082de9c-a6d2-4e01-bc68-385d5d019ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn" Nov 03 22:00:25.241994 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.241975 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1082de9c-a6d2-4e01-bc68-385d5d019ce7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ecc6d17-8n5wn\" (UID: \"1082de9c-a6d2-4e01-bc68-385d5d019ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn" Nov 03 22:00:25.242590 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.242564 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1082de9c-a6d2-4e01-bc68-385d5d019ce7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ecc6d17-8n5wn\" (UID: \"1082de9c-a6d2-4e01-bc68-385d5d019ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn" Nov 03 22:00:25.244253 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.244223 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1082de9c-a6d2-4e01-bc68-385d5d019ce7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ecc6d17-8n5wn\" (UID: \"1082de9c-a6d2-4e01-bc68-385d5d019ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn" Nov 03 22:00:25.263257 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.263226 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhf5h\" (UniqueName: \"kubernetes.io/projected/1082de9c-a6d2-4e01-bc68-385d5d019ce7-kube-api-access-mhf5h\") pod \"maas-api-revoke-sub-5ecc6d17-8n5wn\" (UID: \"1082de9c-a6d2-4e01-bc68-385d5d019ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn" Nov 03 22:00:25.386962 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.386875 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn" Nov 03 22:00:25.685568 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:25.685532 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn"] Nov 03 22:00:25.687191 ip-10-0-131-253 kubenswrapper[5041]: W1103 22:00:25.687160 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1082de9c_a6d2_4e01_bc68_385d5d019ce7.slice/crio-b1d5acb03b96be85a3eb71b424a4a1c3fcd7a02a21a1de2874a669303923bafd WatchSource:0}: Error finding container b1d5acb03b96be85a3eb71b424a4a1c3fcd7a02a21a1de2874a669303923bafd: Status 404 returned error can't find the container with id b1d5acb03b96be85a3eb71b424a4a1c3fcd7a02a21a1de2874a669303923bafd Nov 03 22:00:26.015969 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:26.015925 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn" event={"ID":"1082de9c-a6d2-4e01-bc68-385d5d019ce7","Type":"ContainerStarted","Data":"b1d5acb03b96be85a3eb71b424a4a1c3fcd7a02a21a1de2874a669303923bafd"} Nov 03 22:00:27.023843 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:27.023764 5041 generic.go:356] "Generic (PLEG): container finished" podID="1082de9c-a6d2-4e01-bc68-385d5d019ce7" containerID="a37055562d355e11fee252850eb35e7c608afe431e0ddbabf8dba4248f360e5e" exitCode=0 Nov 03 22:00:27.024216 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:27.023846 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn" event={"ID":"1082de9c-a6d2-4e01-bc68-385d5d019ce7","Type":"ContainerDied","Data":"a37055562d355e11fee252850eb35e7c608afe431e0ddbabf8dba4248f360e5e"} Nov 03 22:00:28.184393 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:28.184365 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn" Nov 03 22:00:28.276180 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:28.276143 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1082de9c-a6d2-4e01-bc68-385d5d019ce7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1082de9c-a6d2-4e01-bc68-385d5d019ce7-maas-api-xks-ca\") pod \"1082de9c-a6d2-4e01-bc68-385d5d019ce7\" (UID: \"1082de9c-a6d2-4e01-bc68-385d5d019ce7\") " Nov 03 22:00:28.276377 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:28.276223 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1082de9c-a6d2-4e01-bc68-385d5d019ce7-kube-api-access-mhf5h\" (UniqueName: \"kubernetes.io/projected/1082de9c-a6d2-4e01-bc68-385d5d019ce7-kube-api-access-mhf5h\") pod \"1082de9c-a6d2-4e01-bc68-385d5d019ce7\" (UID: \"1082de9c-a6d2-4e01-bc68-385d5d019ce7\") " Nov 03 22:00:28.276377 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:28.276314 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1082de9c-a6d2-4e01-bc68-385d5d019ce7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1082de9c-a6d2-4e01-bc68-385d5d019ce7-maas-api-openshift-ca\") pod \"1082de9c-a6d2-4e01-bc68-385d5d019ce7\" (UID: \"1082de9c-a6d2-4e01-bc68-385d5d019ce7\") " Nov 03 22:00:28.276763 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:28.276730 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1082de9c-a6d2-4e01-bc68-385d5d019ce7-maas-api-openshift-ca" pod "1082de9c-a6d2-4e01-bc68-385d5d019ce7" (UID: "1082de9c-a6d2-4e01-bc68-385d5d019ce7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:28.278994 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:28.278962 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1082de9c-a6d2-4e01-bc68-385d5d019ce7-maas-api-xks-ca" pod "1082de9c-a6d2-4e01-bc68-385d5d019ce7" (UID: "1082de9c-a6d2-4e01-bc68-385d5d019ce7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:00:28.279088 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:28.278990 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1082de9c-a6d2-4e01-bc68-385d5d019ce7-kube-api-access-mhf5h" pod "1082de9c-a6d2-4e01-bc68-385d5d019ce7" (UID: "1082de9c-a6d2-4e01-bc68-385d5d019ce7"). InnerVolumeSpecName "kube-api-access-mhf5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:28.377993 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:28.377961 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1082de9c-a6d2-4e01-bc68-385d5d019ce7-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:00:28.377993 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:28.377991 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1082de9c-a6d2-4e01-bc68-385d5d019ce7-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:00:28.377993 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:28.378002 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhf5h\" (UniqueName: \"kubernetes.io/projected/1082de9c-a6d2-4e01-bc68-385d5d019ce7-kube-api-access-mhf5h\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:00:29.037459 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:29.037426 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn" Nov 03 22:00:29.037650 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:29.037463 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn" event={"ID":"1082de9c-a6d2-4e01-bc68-385d5d019ce7","Type":"ContainerDied","Data":"b1d5acb03b96be85a3eb71b424a4a1c3fcd7a02a21a1de2874a669303923bafd"} Nov 03 22:00:29.037650 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:29.037539 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1d5acb03b96be85a3eb71b424a4a1c3fcd7a02a21a1de2874a669303923bafd" Nov 03 22:00:36.226920 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.226883 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs"] Nov 03 22:00:36.227447 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.227428 5041 cpu_manager.go:414] "containerMap: removing container" podUID="1082de9c-a6d2-4e01-bc68-385d5d019ce7" containerName="revoke-keys" Nov 03 22:00:36.227447 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.227449 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1082de9c-a6d2-4e01-bc68-385d5d019ce7" containerName="revoke-keys" Nov 03 22:00:36.227621 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.227485 5041 cpu_manager.go:414] "containerMap: removing container" podUID="6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" containerName="cleanup" Nov 03 22:00:36.227621 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.227494 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" containerName="cleanup" Nov 03 22:00:36.227791 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.227637 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1082de9c-a6d2-4e01-bc68-385d5d019ce7" containerName="revoke-keys" Nov 03 22:00:36.227791 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.227656 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e5b9a71-89d3-4ea7-97dc-5d3cdf857ed6" containerName="cleanup" Nov 03 22:00:36.231262 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.231244 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs" Nov 03 22:00:36.249327 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.249295 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs"] Nov 03 22:00:36.258683 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.258623 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:00:36.258683 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.258636 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 22:00:36.363800 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.363767 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e114d5-f743-4a01-9f89-b793327cbdae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs\" (UID: \"16e114d5-f743-4a01-9f89-b793327cbdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs" Nov 03 22:00:36.363979 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.363952 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmng9\" (UniqueName: \"kubernetes.io/projected/16e114d5-f743-4a01-9f89-b793327cbdae-kube-api-access-rmng9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs\" (UID: \"16e114d5-f743-4a01-9f89-b793327cbdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs" Nov 03 22:00:36.364027 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.364014 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e114d5-f743-4a01-9f89-b793327cbdae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs\" (UID: \"16e114d5-f743-4a01-9f89-b793327cbdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs" Nov 03 22:00:36.465131 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.465092 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e114d5-f743-4a01-9f89-b793327cbdae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs\" (UID: \"16e114d5-f743-4a01-9f89-b793327cbdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs" Nov 03 22:00:36.465323 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.465203 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmng9\" (UniqueName: \"kubernetes.io/projected/16e114d5-f743-4a01-9f89-b793327cbdae-kube-api-access-rmng9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs\" (UID: \"16e114d5-f743-4a01-9f89-b793327cbdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs" Nov 03 22:00:36.465323 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.465243 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e114d5-f743-4a01-9f89-b793327cbdae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs\" (UID: \"16e114d5-f743-4a01-9f89-b793327cbdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs" Nov 03 22:00:36.465850 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.465820 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e114d5-f743-4a01-9f89-b793327cbdae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs\" (UID: \"16e114d5-f743-4a01-9f89-b793327cbdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs" Nov 03 22:00:36.467536 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.467511 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e114d5-f743-4a01-9f89-b793327cbdae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs\" (UID: \"16e114d5-f743-4a01-9f89-b793327cbdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs" Nov 03 22:00:36.485845 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.485773 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmng9\" (UniqueName: \"kubernetes.io/projected/16e114d5-f743-4a01-9f89-b793327cbdae-kube-api-access-rmng9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs\" (UID: \"16e114d5-f743-4a01-9f89-b793327cbdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs" Nov 03 22:00:36.545740 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.545701 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs" Nov 03 22:00:36.842330 ip-10-0-131-253 kubenswrapper[5041]: W1103 22:00:36.842284 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e114d5_f743_4a01_9f89_b793327cbdae.slice/crio-86e7dc0b0f190eee9b0a8d645a5d456b5f6da8432cecf7dd44bd35823d94f335 WatchSource:0}: Error finding container 86e7dc0b0f190eee9b0a8d645a5d456b5f6da8432cecf7dd44bd35823d94f335: Status 404 returned error can't find the container with id 86e7dc0b0f190eee9b0a8d645a5d456b5f6da8432cecf7dd44bd35823d94f335 Nov 03 22:00:36.842640 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:36.842612 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs"] Nov 03 22:00:37.093698 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:37.093567 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs" event={"ID":"16e114d5-f743-4a01-9f89-b793327cbdae","Type":"ContainerStarted","Data":"86e7dc0b0f190eee9b0a8d645a5d456b5f6da8432cecf7dd44bd35823d94f335"} Nov 03 22:00:38.101587 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:38.101542 5041 generic.go:356] "Generic (PLEG): container finished" podID="16e114d5-f743-4a01-9f89-b793327cbdae" containerID="7a3b4e067a4334d232edf667aab0e6ce9f94925f56ca17d2e65f62c679263fe5" exitCode=0 Nov 03 22:00:38.102253 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:38.101611 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs" event={"ID":"16e114d5-f743-4a01-9f89-b793327cbdae","Type":"ContainerDied","Data":"7a3b4e067a4334d232edf667aab0e6ce9f94925f56ca17d2e65f62c679263fe5"} Nov 03 22:00:39.268689 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:39.268634 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs" Nov 03 22:00:39.394332 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:39.394231 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e114d5-f743-4a01-9f89-b793327cbdae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e114d5-f743-4a01-9f89-b793327cbdae-maas-api-openshift-ca\") pod \"16e114d5-f743-4a01-9f89-b793327cbdae\" (UID: \"16e114d5-f743-4a01-9f89-b793327cbdae\") " Nov 03 22:00:39.394517 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:39.394438 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e114d5-f743-4a01-9f89-b793327cbdae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e114d5-f743-4a01-9f89-b793327cbdae-maas-api-xks-ca\") pod \"16e114d5-f743-4a01-9f89-b793327cbdae\" (UID: \"16e114d5-f743-4a01-9f89-b793327cbdae\") " Nov 03 22:00:39.394517 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:39.394474 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e114d5-f743-4a01-9f89-b793327cbdae-kube-api-access-rmng9\" (UniqueName: \"kubernetes.io/projected/16e114d5-f743-4a01-9f89-b793327cbdae-kube-api-access-rmng9\") pod \"16e114d5-f743-4a01-9f89-b793327cbdae\" (UID: \"16e114d5-f743-4a01-9f89-b793327cbdae\") " Nov 03 22:00:39.394723 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:39.394700 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e114d5-f743-4a01-9f89-b793327cbdae-maas-api-openshift-ca" pod "16e114d5-f743-4a01-9f89-b793327cbdae" (UID: "16e114d5-f743-4a01-9f89-b793327cbdae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:00:39.394955 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:39.394936 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e114d5-f743-4a01-9f89-b793327cbdae-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:00:39.396782 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:39.396751 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e114d5-f743-4a01-9f89-b793327cbdae-maas-api-xks-ca" pod "16e114d5-f743-4a01-9f89-b793327cbdae" (UID: "16e114d5-f743-4a01-9f89-b793327cbdae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:00:39.396874 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:39.396783 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e114d5-f743-4a01-9f89-b793327cbdae-kube-api-access-rmng9" pod "16e114d5-f743-4a01-9f89-b793327cbdae" (UID: "16e114d5-f743-4a01-9f89-b793327cbdae"). InnerVolumeSpecName "kube-api-access-rmng9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:00:39.495554 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:39.495514 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e114d5-f743-4a01-9f89-b793327cbdae-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:00:39.495554 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:39.495547 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmng9\" (UniqueName: \"kubernetes.io/projected/16e114d5-f743-4a01-9f89-b793327cbdae-kube-api-access-rmng9\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:00:40.117025 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:40.116988 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs" Nov 03 22:00:40.117025 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:40.117017 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs" event={"ID":"16e114d5-f743-4a01-9f89-b793327cbdae","Type":"ContainerDied","Data":"86e7dc0b0f190eee9b0a8d645a5d456b5f6da8432cecf7dd44bd35823d94f335"} Nov 03 22:00:40.117252 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:40.117068 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86e7dc0b0f190eee9b0a8d645a5d456b5f6da8432cecf7dd44bd35823d94f335" Nov 03 22:00:40.778428 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:00:40.778392 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:00:49.396981 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:49.396946 5041 scope.go:122] "RemoveContainer" containerID="a3819e362a8aa9c847718bc8babe0e4d41b8e3a6a796dd74c9098bd05bfb2b4b" Nov 03 22:00:49.407423 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:49.407395 5041 scope.go:122] "RemoveContainer" containerID="89a36ab5a3babf05141d9026cdc0c754d15c97e8b708136a8e0e2af18f4dc3c5" Nov 03 22:00:49.416293 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:49.416270 5041 scope.go:122] "RemoveContainer" containerID="2ffaa4f6f965b5242143279944cfbbef9e0db4bb9335117d231e6db0450eca05" Nov 03 22:00:50.410981 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:50.410938 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs"] Nov 03 22:00:50.423673 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:50.423621 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3503c0-f29bs"] Nov 03 22:00:50.620844 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:50.620809 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e114d5-f743-4a01-9f89-b793327cbdae" path="/var/lib/kubelet/pods/16e114d5-f743-4a01-9f89-b793327cbdae/volumes" Nov 03 22:00:56.839220 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:56.839179 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j"] Nov 03 22:00:56.839807 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:00:56.839448 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" podUID="b2b07d66-ac37-40a9-8a39-7177f2350915" containerName="istio-proxy" containerID="cri-o://cd6a814acbdd47fdc1b712446074b916fce0df4bf9614f93f850cda16359afb8" gracePeriod=30 Nov 03 22:01:02.104603 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.104532 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 22:01:02.139860 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.139819 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-workload-certs\") pod \"b2b07d66-ac37-40a9-8a39-7177f2350915\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " Nov 03 22:01:02.140039 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.139888 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2b07d66-ac37-40a9-8a39-7177f2350915-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b07d66-ac37-40a9-8a39-7177f2350915-istiod-ca-cert\") pod \"b2b07d66-ac37-40a9-8a39-7177f2350915\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " Nov 03 22:01:02.140039 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.139910 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-token\" (UniqueName: \"kubernetes.io/projected/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-token\") pod \"b2b07d66-ac37-40a9-8a39-7177f2350915\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " Nov 03 22:01:02.140039 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.139978 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-workload-socket\") pod \"b2b07d66-ac37-40a9-8a39-7177f2350915\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " Nov 03 22:01:02.140039 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.140023 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-podinfo\") pod \"b2b07d66-ac37-40a9-8a39-7177f2350915\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " Nov 03 22:01:02.140286 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.140070 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-envoy\") pod \"b2b07d66-ac37-40a9-8a39-7177f2350915\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " Nov 03 22:01:02.140286 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.140126 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-data\") pod \"b2b07d66-ac37-40a9-8a39-7177f2350915\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " Nov 03 22:01:02.140286 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.140161 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-workload-certs" pod "b2b07d66-ac37-40a9-8a39-7177f2350915" (UID: "b2b07d66-ac37-40a9-8a39-7177f2350915"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:01:02.140286 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.140169 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b07d66-ac37-40a9-8a39-7177f2350915-kube-api-access-m6f2f\" (UniqueName: \"kubernetes.io/projected/b2b07d66-ac37-40a9-8a39-7177f2350915-kube-api-access-m6f2f\") pod \"b2b07d66-ac37-40a9-8a39-7177f2350915\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " Nov 03 22:01:02.140505 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.140304 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-credential-socket\") pod \"b2b07d66-ac37-40a9-8a39-7177f2350915\" (UID: \"b2b07d66-ac37-40a9-8a39-7177f2350915\") " Nov 03 22:01:02.140505 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.140331 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-workload-socket" pod "b2b07d66-ac37-40a9-8a39-7177f2350915" (UID: "b2b07d66-ac37-40a9-8a39-7177f2350915"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:01:02.140505 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.140362 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b07d66-ac37-40a9-8a39-7177f2350915-istiod-ca-cert" pod "b2b07d66-ac37-40a9-8a39-7177f2350915" (UID: "b2b07d66-ac37-40a9-8a39-7177f2350915"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:01:02.140719 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.140687 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-data" pod "b2b07d66-ac37-40a9-8a39-7177f2350915" (UID: "b2b07d66-ac37-40a9-8a39-7177f2350915"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:01:02.140795 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.140747 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-credential-socket" pod "b2b07d66-ac37-40a9-8a39-7177f2350915" (UID: "b2b07d66-ac37-40a9-8a39-7177f2350915"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:01:02.140795 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.140784 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:01:02.140905 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.140806 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2b07d66-ac37-40a9-8a39-7177f2350915-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:01:02.140905 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.140819 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:01:02.142422 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.142399 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-envoy" pod "b2b07d66-ac37-40a9-8a39-7177f2350915" (UID: "b2b07d66-ac37-40a9-8a39-7177f2350915"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:01:02.142514 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.142425 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b07d66-ac37-40a9-8a39-7177f2350915-kube-api-access-m6f2f" pod "b2b07d66-ac37-40a9-8a39-7177f2350915" (UID: "b2b07d66-ac37-40a9-8a39-7177f2350915"). InnerVolumeSpecName "kube-api-access-m6f2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:01:02.142707 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.142688 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-podinfo" pod "b2b07d66-ac37-40a9-8a39-7177f2350915" (UID: "b2b07d66-ac37-40a9-8a39-7177f2350915"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:01:02.142764 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.142729 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-token" pod "b2b07d66-ac37-40a9-8a39-7177f2350915" (UID: "b2b07d66-ac37-40a9-8a39-7177f2350915"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:01:02.242208 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.242173 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:01:02.242208 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.242202 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:01:02.242208 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.242214 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:01:02.242464 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.242222 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:01:02.242464 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.242231 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6f2f\" (UniqueName: \"kubernetes.io/projected/b2b07d66-ac37-40a9-8a39-7177f2350915-kube-api-access-m6f2f\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:01:02.242464 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.242241 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2b07d66-ac37-40a9-8a39-7177f2350915-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:01:02.269948 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.269914 5041 generic.go:356] "Generic (PLEG): container finished" podID="b2b07d66-ac37-40a9-8a39-7177f2350915" containerID="cd6a814acbdd47fdc1b712446074b916fce0df4bf9614f93f850cda16359afb8" exitCode=0 Nov 03 22:01:02.270127 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.269964 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" Nov 03 22:01:02.270127 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.270007 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" event={"ID":"b2b07d66-ac37-40a9-8a39-7177f2350915","Type":"ContainerDied","Data":"cd6a814acbdd47fdc1b712446074b916fce0df4bf9614f93f850cda16359afb8"} Nov 03 22:01:02.270127 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.270074 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j" event={"ID":"b2b07d66-ac37-40a9-8a39-7177f2350915","Type":"ContainerDied","Data":"5f347357c988874bc1d2751b4dc6c9df0bef2b3a194e271d577cda88997d0093"} Nov 03 22:01:02.270127 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.270103 5041 scope.go:122] "RemoveContainer" containerID="cd6a814acbdd47fdc1b712446074b916fce0df4bf9614f93f850cda16359afb8" Nov 03 22:01:02.281546 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.281529 5041 scope.go:122] "RemoveContainer" containerID="9545b667731c82179832530295c47d95ed02d2fc23efb871d44e88365dab933b" Nov 03 22:01:02.291131 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.291110 5041 scope.go:122] "RemoveContainer" containerID="cd6a814acbdd47fdc1b712446074b916fce0df4bf9614f93f850cda16359afb8" Nov 03 22:01:02.291471 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:01:02.291450 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd6a814acbdd47fdc1b712446074b916fce0df4bf9614f93f850cda16359afb8\": container with ID starting with cd6a814acbdd47fdc1b712446074b916fce0df4bf9614f93f850cda16359afb8 not found: ID does not exist" containerID="cd6a814acbdd47fdc1b712446074b916fce0df4bf9614f93f850cda16359afb8" Nov 03 22:01:02.291536 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.291482 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6a814acbdd47fdc1b712446074b916fce0df4bf9614f93f850cda16359afb8"} err="failed to get container status \"cd6a814acbdd47fdc1b712446074b916fce0df4bf9614f93f850cda16359afb8\": rpc error: code = NotFound desc = could not find container \"cd6a814acbdd47fdc1b712446074b916fce0df4bf9614f93f850cda16359afb8\": container with ID starting with cd6a814acbdd47fdc1b712446074b916fce0df4bf9614f93f850cda16359afb8 not found: ID does not exist" Nov 03 22:01:02.291536 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.291513 5041 scope.go:122] "RemoveContainer" containerID="9545b667731c82179832530295c47d95ed02d2fc23efb871d44e88365dab933b" Nov 03 22:01:02.291825 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:01:02.291808 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9545b667731c82179832530295c47d95ed02d2fc23efb871d44e88365dab933b\": container with ID starting with 9545b667731c82179832530295c47d95ed02d2fc23efb871d44e88365dab933b not found: ID does not exist" containerID="9545b667731c82179832530295c47d95ed02d2fc23efb871d44e88365dab933b" Nov 03 22:01:02.291880 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.291830 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9545b667731c82179832530295c47d95ed02d2fc23efb871d44e88365dab933b"} err="failed to get container status \"9545b667731c82179832530295c47d95ed02d2fc23efb871d44e88365dab933b\": rpc error: code = NotFound desc = could not find container \"9545b667731c82179832530295c47d95ed02d2fc23efb871d44e88365dab933b\": container with ID starting with 9545b667731c82179832530295c47d95ed02d2fc23efb871d44e88365dab933b not found: ID does not exist" Nov 03 22:01:02.326780 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.326742 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j"] Nov 03 22:01:02.331925 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.331901 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3503c0-openshift-default-6dd6b74f659vc7j"] Nov 03 22:01:02.624008 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:02.623973 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2b07d66-ac37-40a9-8a39-7177f2350915" path="/var/lib/kubelet/pods/b2b07d66-ac37-40a9-8a39-7177f2350915/volumes" Nov 03 22:01:10.778348 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:01:10.778300 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:01:17.938321 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:17.938283 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p"] Nov 03 22:01:17.938848 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:17.938831 5041 cpu_manager.go:414] "containerMap: removing container" podUID="b2b07d66-ac37-40a9-8a39-7177f2350915" containerName="istio-proxy" Nov 03 22:01:17.938902 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:17.938849 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b07d66-ac37-40a9-8a39-7177f2350915" containerName="istio-proxy" Nov 03 22:01:17.938902 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:17.938882 5041 cpu_manager.go:414] "containerMap: removing container" podUID="16e114d5-f743-4a01-9f89-b793327cbdae" containerName="revoke-keys" Nov 03 22:01:17.938902 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:17.938888 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e114d5-f743-4a01-9f89-b793327cbdae" containerName="revoke-keys" Nov 03 22:01:17.939004 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:17.938960 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e114d5-f743-4a01-9f89-b793327cbdae" containerName="revoke-keys" Nov 03 22:01:17.939004 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:17.938972 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b07d66-ac37-40a9-8a39-7177f2350915" containerName="istio-proxy" Nov 03 22:01:17.942790 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:17.942762 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:17.961393 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:17.961347 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p"] Nov 03 22:01:17.970759 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:17.970716 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-5efb7ae3-kserve-self-signed-certs\"" Nov 03 22:01:18.078947 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.078913 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmstc\" (UniqueName: \"kubernetes.io/projected/cf5183c5-8f2b-4406-913f-0383b83b694c-kube-api-access-jmstc\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.079150 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.078984 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5183c5-8f2b-4406-913f-0383b83b694c-tls-certs\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.079150 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.079049 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-dshm\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.079150 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.079085 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-home\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.079150 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.079111 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-tmp-dir\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.079323 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.079240 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-model-cache\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.180544 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.180500 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-tmp-dir\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.180751 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.180556 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-model-cache\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.180751 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.180609 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmstc\" (UniqueName: \"kubernetes.io/projected/cf5183c5-8f2b-4406-913f-0383b83b694c-kube-api-access-jmstc\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.180751 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.180653 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5183c5-8f2b-4406-913f-0383b83b694c-tls-certs\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.180751 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.180713 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-dshm\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.181013 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.180754 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-home\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.181102 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.181079 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-tmp-dir\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.181160 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.181110 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-model-cache\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.181203 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.181170 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-home\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.183085 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.183057 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-dshm\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.183384 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.183364 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5183c5-8f2b-4406-913f-0383b83b694c-tls-certs\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.200974 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.200900 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmstc\" (UniqueName: \"kubernetes.io/projected/cf5183c5-8f2b-4406-913f-0383b83b694c-kube-api-access-jmstc\") pod \"e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.260709 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.260654 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:18.568098 ip-10-0-131-253 kubenswrapper[5041]: W1103 22:01:18.568066 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf5183c5_8f2b_4406_913f_0383b83b694c.slice/crio-c2b59251860eb2e5f34002e2846434eded13b1fa069c7237b999b849d2a100a1 WatchSource:0}: Error finding container c2b59251860eb2e5f34002e2846434eded13b1fa069c7237b999b849d2a100a1: Status 404 returned error can't find the container with id c2b59251860eb2e5f34002e2846434eded13b1fa069c7237b999b849d2a100a1 Nov 03 22:01:18.568233 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:18.568200 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p"] Nov 03 22:01:19.465065 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:19.465024 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" event={"ID":"cf5183c5-8f2b-4406-913f-0383b83b694c","Type":"ContainerStarted","Data":"cb50e438daa059d70c31d837ccfad8d443bed36e650cc5fd8399787830c52a05"} Nov 03 22:01:19.465522 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:19.465079 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" event={"ID":"cf5183c5-8f2b-4406-913f-0383b83b694c","Type":"ContainerStarted","Data":"c2b59251860eb2e5f34002e2846434eded13b1fa069c7237b999b849d2a100a1"} Nov 03 22:01:19.506923 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:19.506867 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" podStartSLOduration=2.5068519609999997 podStartE2EDuration="2.506851961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:01:19.50536172 +0000 UTC m=+1895.012590018" watchObservedRunningTime="2026-11-03 22:01:19.506851961 +0000 UTC m=+1895.014080240" Nov 03 22:01:20.261628 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:20.261589 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:24.191822 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:24.189846 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-wc8rh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:01:24.191822 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:24.189936 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" podUID="44bc42a8-430e-44ec-b717-d1a2436968f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:01:24.235915 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:24.235766 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-8b9jr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:01:24.235915 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:24.235841 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" podUID="69cc2bab-a8e9-4a4a-a114-88663140640d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:01:28.261106 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:28.261068 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:28.273629 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:28.273596 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:28.568567 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:28.568471 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:01:35.027267 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:35.027221 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m"] Nov 03 22:01:35.040747 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:35.040707 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feed85b5-4kr5m"] Nov 03 22:01:36.623187 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.623145 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b050a83e-b8e5-429e-a0c5-419f5e20c839" path="/var/lib/kubelet/pods/b050a83e-b8e5-429e-a0c5-419f5e20c839/volumes" Nov 03 22:01:36.741179 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.741138 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l"] Nov 03 22:01:36.742205 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.742160 5041 cpu_manager.go:414] "containerMap: removing container" podUID="b2b07d66-ac37-40a9-8a39-7177f2350915" containerName="istio-proxy" Nov 03 22:01:36.742395 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.742377 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b07d66-ac37-40a9-8a39-7177f2350915" containerName="istio-proxy" Nov 03 22:01:36.742717 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.742699 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b07d66-ac37-40a9-8a39-7177f2350915" containerName="istio-proxy" Nov 03 22:01:36.747110 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.747070 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" Nov 03 22:01:36.763063 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.762193 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l"] Nov 03 22:01:36.763972 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.763948 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 22:01:36.765888 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.765792 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:01:36.845206 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.845149 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgm7x\" (UniqueName: \"kubernetes.io/projected/de52a474-5c45-4c1e-9a1e-c204a154d00c-kube-api-access-fgm7x\") pod \"maas-api-revoke-sub-452b6d7b-nmg5l\" (UID: \"de52a474-5c45-4c1e-9a1e-c204a154d00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" Nov 03 22:01:36.845429 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.845340 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de52a474-5c45-4c1e-9a1e-c204a154d00c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nmg5l\" (UID: \"de52a474-5c45-4c1e-9a1e-c204a154d00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" Nov 03 22:01:36.845536 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.845510 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de52a474-5c45-4c1e-9a1e-c204a154d00c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nmg5l\" (UID: \"de52a474-5c45-4c1e-9a1e-c204a154d00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" Nov 03 22:01:36.946595 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.946473 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de52a474-5c45-4c1e-9a1e-c204a154d00c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nmg5l\" (UID: \"de52a474-5c45-4c1e-9a1e-c204a154d00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" Nov 03 22:01:36.946820 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.946620 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de52a474-5c45-4c1e-9a1e-c204a154d00c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nmg5l\" (UID: \"de52a474-5c45-4c1e-9a1e-c204a154d00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" Nov 03 22:01:36.946820 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.946686 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgm7x\" (UniqueName: \"kubernetes.io/projected/de52a474-5c45-4c1e-9a1e-c204a154d00c-kube-api-access-fgm7x\") pod \"maas-api-revoke-sub-452b6d7b-nmg5l\" (UID: \"de52a474-5c45-4c1e-9a1e-c204a154d00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" Nov 03 22:01:36.947469 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.947431 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de52a474-5c45-4c1e-9a1e-c204a154d00c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nmg5l\" (UID: \"de52a474-5c45-4c1e-9a1e-c204a154d00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" Nov 03 22:01:36.949876 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.949843 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de52a474-5c45-4c1e-9a1e-c204a154d00c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nmg5l\" (UID: \"de52a474-5c45-4c1e-9a1e-c204a154d00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" Nov 03 22:01:36.970395 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:36.970358 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgm7x\" (UniqueName: \"kubernetes.io/projected/de52a474-5c45-4c1e-9a1e-c204a154d00c-kube-api-access-fgm7x\") pod \"maas-api-revoke-sub-452b6d7b-nmg5l\" (UID: \"de52a474-5c45-4c1e-9a1e-c204a154d00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" Nov 03 22:01:37.081900 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:37.081854 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" Nov 03 22:01:37.479294 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:37.479079 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l"] Nov 03 22:01:37.621552 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:37.621505 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" event={"ID":"de52a474-5c45-4c1e-9a1e-c204a154d00c","Type":"ContainerStarted","Data":"6992a92d5a6e2944d723d6a295dc01b7aa1469e00504cbbbd261e7e3ef1bf834"} Nov 03 22:01:38.761315 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:38.760304 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" event={"ID":"de52a474-5c45-4c1e-9a1e-c204a154d00c","Type":"ContainerStarted","Data":"af556f47d204d183d907933878f87b6cd4a890f916011432d77e16ef3d2405ac"} Nov 03 22:01:38.806847 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:38.806786 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" podStartSLOduration=2.806765655 podStartE2EDuration="2.806765655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:01:38.80406045 +0000 UTC m=+1914.311288730" watchObservedRunningTime="2026-11-03 22:01:38.806765655 +0000 UTC m=+1914.313993937" Nov 03 22:01:39.239928 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:39.239851 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-8b9jr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:01:39.239928 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:39.239920 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-8b9jr" podUID="69cc2bab-a8e9-4a4a-a114-88663140640d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:01:39.240444 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:39.240415 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-wc8rh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:01:39.240562 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:39.240466 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-wc8rh" podUID="44bc42a8-430e-44ec-b717-d1a2436968f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:01:39.260798 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:01:39.260753 5041 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde52a474_5c45_4c1e_9a1e_c204a154d00c.slice/crio-conmon-af556f47d204d183d907933878f87b6cd4a890f916011432d77e16ef3d2405ac.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:01:39.601882 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:39.601778 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6kgrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:01:39.602173 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:39.602140 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6kgrf" podUID="bd662fa8-196b-4cad-b414-1bc2dbd68aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:01:39.782112 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:39.782069 5041 generic.go:356] "Generic (PLEG): container finished" podID="de52a474-5c45-4c1e-9a1e-c204a154d00c" containerID="af556f47d204d183d907933878f87b6cd4a890f916011432d77e16ef3d2405ac" exitCode=0 Nov 03 22:01:39.782609 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:39.782227 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" event={"ID":"de52a474-5c45-4c1e-9a1e-c204a154d00c","Type":"ContainerDied","Data":"af556f47d204d183d907933878f87b6cd4a890f916011432d77e16ef3d2405ac"} Nov 03 22:01:40.776825 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:01:40.776780 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:01:40.936754 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:40.936727 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" Nov 03 22:01:41.036593 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:41.036495 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de52a474-5c45-4c1e-9a1e-c204a154d00c-kube-api-access-fgm7x\" (UniqueName: \"kubernetes.io/projected/de52a474-5c45-4c1e-9a1e-c204a154d00c-kube-api-access-fgm7x\") pod \"de52a474-5c45-4c1e-9a1e-c204a154d00c\" (UID: \"de52a474-5c45-4c1e-9a1e-c204a154d00c\") " Nov 03 22:01:41.036593 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:41.036591 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de52a474-5c45-4c1e-9a1e-c204a154d00c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de52a474-5c45-4c1e-9a1e-c204a154d00c-maas-api-xks-ca\") pod \"de52a474-5c45-4c1e-9a1e-c204a154d00c\" (UID: \"de52a474-5c45-4c1e-9a1e-c204a154d00c\") " Nov 03 22:01:41.036881 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:41.036696 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de52a474-5c45-4c1e-9a1e-c204a154d00c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de52a474-5c45-4c1e-9a1e-c204a154d00c-maas-api-openshift-ca\") pod \"de52a474-5c45-4c1e-9a1e-c204a154d00c\" (UID: \"de52a474-5c45-4c1e-9a1e-c204a154d00c\") " Nov 03 22:01:41.037109 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:41.037082 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de52a474-5c45-4c1e-9a1e-c204a154d00c-maas-api-openshift-ca" pod "de52a474-5c45-4c1e-9a1e-c204a154d00c" (UID: "de52a474-5c45-4c1e-9a1e-c204a154d00c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:01:41.038757 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:41.038726 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de52a474-5c45-4c1e-9a1e-c204a154d00c-kube-api-access-fgm7x" pod "de52a474-5c45-4c1e-9a1e-c204a154d00c" (UID: "de52a474-5c45-4c1e-9a1e-c204a154d00c"). InnerVolumeSpecName "kube-api-access-fgm7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:01:41.038757 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:41.038747 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de52a474-5c45-4c1e-9a1e-c204a154d00c-maas-api-xks-ca" pod "de52a474-5c45-4c1e-9a1e-c204a154d00c" (UID: "de52a474-5c45-4c1e-9a1e-c204a154d00c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:01:41.138437 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:41.138405 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de52a474-5c45-4c1e-9a1e-c204a154d00c-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:01:41.138437 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:41.138434 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgm7x\" (UniqueName: \"kubernetes.io/projected/de52a474-5c45-4c1e-9a1e-c204a154d00c-kube-api-access-fgm7x\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:01:41.138437 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:41.138445 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de52a474-5c45-4c1e-9a1e-c204a154d00c-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:01:41.796208 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:41.796173 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" event={"ID":"de52a474-5c45-4c1e-9a1e-c204a154d00c","Type":"ContainerDied","Data":"6992a92d5a6e2944d723d6a295dc01b7aa1469e00504cbbbd261e7e3ef1bf834"} Nov 03 22:01:41.796208 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:41.796191 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l" Nov 03 22:01:41.796433 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:41.796227 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6992a92d5a6e2944d723d6a295dc01b7aa1469e00504cbbbd261e7e3ef1bf834" Nov 03 22:01:49.453502 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:49.453465 5041 scope.go:122] "RemoveContainer" containerID="4bf4abc1d36d0860d73eb1e164a051cb05816166268a64a88dfef1c366ba09d7" Nov 03 22:01:49.463699 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:49.463673 5041 scope.go:122] "RemoveContainer" containerID="eae8172444daeecec585d259d53c71791c1af8ba1679e52c32b81c49987e2686" Nov 03 22:01:53.819828 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:53.805674 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p"] Nov 03 22:01:58.829868 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:58.813033 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk"] Nov 03 22:01:58.831295 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:58.831259 5041 cpu_manager.go:414] "containerMap: removing container" podUID="de52a474-5c45-4c1e-9a1e-c204a154d00c" containerName="revoke-keys" Nov 03 22:01:58.831483 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:58.831466 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de52a474-5c45-4c1e-9a1e-c204a154d00c" containerName="revoke-keys" Nov 03 22:01:58.831833 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:58.831813 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="de52a474-5c45-4c1e-9a1e-c204a154d00c" containerName="revoke-keys" Nov 03 22:01:58.842907 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:58.842875 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" Nov 03 22:01:58.874798 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:58.864600 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk"] Nov 03 22:01:58.894738 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:58.875675 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8brb\" (UniqueName: \"kubernetes.io/projected/6715bed2-8392-4ccb-aa31-ee9690099b30-kube-api-access-b8brb\") pod \"maas-api-revoke-sub-3e31a506-cgmkk\" (UID: \"6715bed2-8392-4ccb-aa31-ee9690099b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" Nov 03 22:01:58.967509 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:58.966760 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 22:01:58.967776 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:58.967027 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:01:58.967874 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:58.967539 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6715bed2-8392-4ccb-aa31-ee9690099b30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-cgmkk\" (UID: \"6715bed2-8392-4ccb-aa31-ee9690099b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" Nov 03 22:01:58.967934 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:58.967868 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6715bed2-8392-4ccb-aa31-ee9690099b30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-cgmkk\" (UID: \"6715bed2-8392-4ccb-aa31-ee9690099b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" Nov 03 22:01:59.069422 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:59.069388 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6715bed2-8392-4ccb-aa31-ee9690099b30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-cgmkk\" (UID: \"6715bed2-8392-4ccb-aa31-ee9690099b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" Nov 03 22:01:59.069613 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:59.069443 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6715bed2-8392-4ccb-aa31-ee9690099b30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-cgmkk\" (UID: \"6715bed2-8392-4ccb-aa31-ee9690099b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" Nov 03 22:01:59.069613 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:59.069559 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8brb\" (UniqueName: \"kubernetes.io/projected/6715bed2-8392-4ccb-aa31-ee9690099b30-kube-api-access-b8brb\") pod \"maas-api-revoke-sub-3e31a506-cgmkk\" (UID: \"6715bed2-8392-4ccb-aa31-ee9690099b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" Nov 03 22:01:59.070609 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:59.070579 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6715bed2-8392-4ccb-aa31-ee9690099b30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-cgmkk\" (UID: \"6715bed2-8392-4ccb-aa31-ee9690099b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" Nov 03 22:01:59.072460 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:59.072436 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6715bed2-8392-4ccb-aa31-ee9690099b30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-cgmkk\" (UID: \"6715bed2-8392-4ccb-aa31-ee9690099b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" Nov 03 22:01:59.092363 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:59.092297 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8brb\" (UniqueName: \"kubernetes.io/projected/6715bed2-8392-4ccb-aa31-ee9690099b30-kube-api-access-b8brb\") pod \"maas-api-revoke-sub-3e31a506-cgmkk\" (UID: \"6715bed2-8392-4ccb-aa31-ee9690099b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" Nov 03 22:01:59.321693 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:59.321639 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" Nov 03 22:01:59.626439 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:01:59.626401 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk"] Nov 03 22:01:59.626914 ip-10-0-131-253 kubenswrapper[5041]: W1103 22:01:59.626890 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6715bed2_8392_4ccb_aa31_ee9690099b30.slice/crio-3f60a98ff21c5a6674072efed3bd233bdf44e5719b8d010414abbf1a1de9ac8e WatchSource:0}: Error finding container 3f60a98ff21c5a6674072efed3bd233bdf44e5719b8d010414abbf1a1de9ac8e: Status 404 returned error can't find the container with id 3f60a98ff21c5a6674072efed3bd233bdf44e5719b8d010414abbf1a1de9ac8e Nov 03 22:02:00.017026 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:00.016984 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" event={"ID":"6715bed2-8392-4ccb-aa31-ee9690099b30","Type":"ContainerStarted","Data":"d710cb0f04a163dd3d55b4f0ca78a1a9818e00b56ef65250a3af7800d3cfdf80"} Nov 03 22:02:00.017490 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:00.017065 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" event={"ID":"6715bed2-8392-4ccb-aa31-ee9690099b30","Type":"ContainerStarted","Data":"3f60a98ff21c5a6674072efed3bd233bdf44e5719b8d010414abbf1a1de9ac8e"} Nov 03 22:02:00.057373 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:00.057318 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" podStartSLOduration=2.057303476 podStartE2EDuration="2.057303476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:00.057268844 +0000 UTC m=+1935.564497125" watchObservedRunningTime="2026-11-03 22:02:00.057303476 +0000 UTC m=+1935.564531756" Nov 03 22:02:01.025748 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:01.025711 5041 generic.go:356] "Generic (PLEG): container finished" podID="6715bed2-8392-4ccb-aa31-ee9690099b30" containerID="d710cb0f04a163dd3d55b4f0ca78a1a9818e00b56ef65250a3af7800d3cfdf80" exitCode=0 Nov 03 22:02:01.026192 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:01.025800 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" event={"ID":"6715bed2-8392-4ccb-aa31-ee9690099b30","Type":"ContainerDied","Data":"d710cb0f04a163dd3d55b4f0ca78a1a9818e00b56ef65250a3af7800d3cfdf80"} Nov 03 22:02:02.193206 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:02.193175 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" Nov 03 22:02:02.201496 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:02.201471 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6715bed2-8392-4ccb-aa31-ee9690099b30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6715bed2-8392-4ccb-aa31-ee9690099b30-maas-api-openshift-ca\") pod \"6715bed2-8392-4ccb-aa31-ee9690099b30\" (UID: \"6715bed2-8392-4ccb-aa31-ee9690099b30\") " Nov 03 22:02:02.201594 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:02.201507 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6715bed2-8392-4ccb-aa31-ee9690099b30-kube-api-access-b8brb\" (UniqueName: \"kubernetes.io/projected/6715bed2-8392-4ccb-aa31-ee9690099b30-kube-api-access-b8brb\") pod \"6715bed2-8392-4ccb-aa31-ee9690099b30\" (UID: \"6715bed2-8392-4ccb-aa31-ee9690099b30\") " Nov 03 22:02:02.201639 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:02.201626 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6715bed2-8392-4ccb-aa31-ee9690099b30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6715bed2-8392-4ccb-aa31-ee9690099b30-maas-api-xks-ca\") pod \"6715bed2-8392-4ccb-aa31-ee9690099b30\" (UID: \"6715bed2-8392-4ccb-aa31-ee9690099b30\") " Nov 03 22:02:02.201916 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:02.201895 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6715bed2-8392-4ccb-aa31-ee9690099b30-maas-api-openshift-ca" pod "6715bed2-8392-4ccb-aa31-ee9690099b30" (UID: "6715bed2-8392-4ccb-aa31-ee9690099b30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:02:02.202033 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:02.202019 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6715bed2-8392-4ccb-aa31-ee9690099b30-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:02:02.203763 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:02.203738 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6715bed2-8392-4ccb-aa31-ee9690099b30-kube-api-access-b8brb" pod "6715bed2-8392-4ccb-aa31-ee9690099b30" (UID: "6715bed2-8392-4ccb-aa31-ee9690099b30"). InnerVolumeSpecName "kube-api-access-b8brb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:02:02.203763 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:02.203753 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6715bed2-8392-4ccb-aa31-ee9690099b30-maas-api-xks-ca" pod "6715bed2-8392-4ccb-aa31-ee9690099b30" (UID: "6715bed2-8392-4ccb-aa31-ee9690099b30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:02:02.303178 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:02.303092 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8brb\" (UniqueName: \"kubernetes.io/projected/6715bed2-8392-4ccb-aa31-ee9690099b30-kube-api-access-b8brb\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:02:02.303178 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:02.303119 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6715bed2-8392-4ccb-aa31-ee9690099b30-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:02:03.039814 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:03.039780 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" event={"ID":"6715bed2-8392-4ccb-aa31-ee9690099b30","Type":"ContainerDied","Data":"3f60a98ff21c5a6674072efed3bd233bdf44e5719b8d010414abbf1a1de9ac8e"} Nov 03 22:02:03.039992 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:03.039828 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f60a98ff21c5a6674072efed3bd233bdf44e5719b8d010414abbf1a1de9ac8e" Nov 03 22:02:03.039992 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:03.039836 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk" Nov 03 22:02:06.027918 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:06.027883 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp"] Nov 03 22:02:06.041656 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:06.041624 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vsjfp"] Nov 03 22:02:06.620325 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:06.620287 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c7e1963-bf7d-47ae-acf4-253d651e9133" path="/var/lib/kubelet/pods/8c7e1963-bf7d-47ae-acf4-253d651e9133/volumes" Nov 03 22:02:08.912007 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:08.911963 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" podUID="cf5183c5-8f2b-4406-913f-0383b83b694c" containerName="main" containerID="cri-o://cb50e438daa059d70c31d837ccfad8d443bed36e650cc5fd8399787830c52a05" gracePeriod=45 Nov 03 22:02:09.082640 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.080344 5041 generic.go:356] "Generic (PLEG): container finished" podID="cf5183c5-8f2b-4406-913f-0383b83b694c" containerID="cb50e438daa059d70c31d837ccfad8d443bed36e650cc5fd8399787830c52a05" exitCode=0 Nov 03 22:02:09.082640 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.080489 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" event={"ID":"cf5183c5-8f2b-4406-913f-0383b83b694c","Type":"ContainerDied","Data":"cb50e438daa059d70c31d837ccfad8d443bed36e650cc5fd8399787830c52a05"} Nov 03 22:02:09.174365 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.174306 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:02:09.280098 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.280054 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-model-cache\") pod \"cf5183c5-8f2b-4406-913f-0383b83b694c\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " Nov 03 22:02:09.280300 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.280181 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf5183c5-8f2b-4406-913f-0383b83b694c-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5183c5-8f2b-4406-913f-0383b83b694c-tls-certs\") pod \"cf5183c5-8f2b-4406-913f-0383b83b694c\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " Nov 03 22:02:09.280300 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.280231 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-tmp-dir\") pod \"cf5183c5-8f2b-4406-913f-0383b83b694c\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " Nov 03 22:02:09.280443 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.280300 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-dshm\") pod \"cf5183c5-8f2b-4406-913f-0383b83b694c\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " Nov 03 22:02:09.280443 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.280314 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-model-cache" pod "cf5183c5-8f2b-4406-913f-0383b83b694c" (UID: "cf5183c5-8f2b-4406-913f-0383b83b694c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:02:09.280443 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.280351 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5183c5-8f2b-4406-913f-0383b83b694c-kube-api-access-jmstc\" (UniqueName: \"kubernetes.io/projected/cf5183c5-8f2b-4406-913f-0383b83b694c-kube-api-access-jmstc\") pod \"cf5183c5-8f2b-4406-913f-0383b83b694c\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " Nov 03 22:02:09.280443 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.280373 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-home\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-home\") pod \"cf5183c5-8f2b-4406-913f-0383b83b694c\" (UID: \"cf5183c5-8f2b-4406-913f-0383b83b694c\") " Nov 03 22:02:09.280724 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.280706 5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-model-cache\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:02:09.280976 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.280950 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-home" pod "cf5183c5-8f2b-4406-913f-0383b83b694c" (UID: "cf5183c5-8f2b-4406-913f-0383b83b694c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:02:09.281282 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.281210 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-tmp-dir" pod "cf5183c5-8f2b-4406-913f-0383b83b694c" (UID: "cf5183c5-8f2b-4406-913f-0383b83b694c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:02:09.282790 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.282763 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5183c5-8f2b-4406-913f-0383b83b694c-tls-certs" pod "cf5183c5-8f2b-4406-913f-0383b83b694c" (UID: "cf5183c5-8f2b-4406-913f-0383b83b694c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:02:09.282877 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.282785 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5183c5-8f2b-4406-913f-0383b83b694c-kube-api-access-jmstc" pod "cf5183c5-8f2b-4406-913f-0383b83b694c" (UID: "cf5183c5-8f2b-4406-913f-0383b83b694c"). InnerVolumeSpecName "kube-api-access-jmstc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:02:09.282877 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.282845 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-dshm" pod "cf5183c5-8f2b-4406-913f-0383b83b694c" (UID: "cf5183c5-8f2b-4406-913f-0383b83b694c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:02:09.381803 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.381767 5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5183c5-8f2b-4406-913f-0383b83b694c-tls-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:02:09.381803 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.381798 5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-tmp-dir\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:02:09.381803 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.381810 5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-dshm\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:02:09.382039 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.381822 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmstc\" (UniqueName: \"kubernetes.io/projected/cf5183c5-8f2b-4406-913f-0383b83b694c-kube-api-access-jmstc\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:02:09.382039 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:09.381833 5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5183c5-8f2b-4406-913f-0383b83b694c-home\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:02:10.097196 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:10.097162 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" event={"ID":"cf5183c5-8f2b-4406-913f-0383b83b694c","Type":"ContainerDied","Data":"c2b59251860eb2e5f34002e2846434eded13b1fa069c7237b999b849d2a100a1"} Nov 03 22:02:10.097654 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:10.097203 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p" Nov 03 22:02:10.097654 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:10.097213 5041 scope.go:122] "RemoveContainer" containerID="cb50e438daa059d70c31d837ccfad8d443bed36e650cc5fd8399787830c52a05" Nov 03 22:02:10.153204 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:10.153170 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p"] Nov 03 22:02:10.166653 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:10.166615 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-5efb7ae3-kserve-76cb976c69-7sg5p"] Nov 03 22:02:10.688436 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:10.688394 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf5183c5-8f2b-4406-913f-0383b83b694c" path="/var/lib/kubelet/pods/cf5183c5-8f2b-4406-913f-0383b83b694c/volumes" Nov 03 22:02:10.914697 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:02:10.914514 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:02:26.554839 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:26.554785 5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lt4j4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:02:26.555517 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:26.555480 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lt4j4" podUID="9be23a49-ab91-409d-b131-cb4a917a58e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:02:29.177046 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.177004 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4"] Nov 03 22:02:29.177710 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.177688 5041 cpu_manager.go:414] "containerMap: removing container" podUID="cf5183c5-8f2b-4406-913f-0383b83b694c" containerName="main" Nov 03 22:02:29.177819 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.177710 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5183c5-8f2b-4406-913f-0383b83b694c" containerName="main" Nov 03 22:02:29.177819 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.177728 5041 cpu_manager.go:414] "containerMap: removing container" podUID="6715bed2-8392-4ccb-aa31-ee9690099b30" containerName="revoke-keys" Nov 03 22:02:29.177819 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.177736 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6715bed2-8392-4ccb-aa31-ee9690099b30" containerName="revoke-keys" Nov 03 22:02:29.177975 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.177865 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6715bed2-8392-4ccb-aa31-ee9690099b30" containerName="revoke-keys" Nov 03 22:02:29.177975 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.177891 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5183c5-8f2b-4406-913f-0383b83b694c" containerName="main" Nov 03 22:02:29.182304 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.182283 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" Nov 03 22:02:29.198077 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.198049 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4"] Nov 03 22:02:29.208471 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.208444 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:02:29.208599 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.208565 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 22:02:29.272220 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.272188 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2454f93f-f6f2-43bc-8e55-c4974ac964ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56327c97-z4lt4\" (UID: \"2454f93f-f6f2-43bc-8e55-c4974ac964ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" Nov 03 22:02:29.272421 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.272257 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2454f93f-f6f2-43bc-8e55-c4974ac964ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56327c97-z4lt4\" (UID: \"2454f93f-f6f2-43bc-8e55-c4974ac964ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" Nov 03 22:02:29.272421 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.272355 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w992q\" (UniqueName: \"kubernetes.io/projected/2454f93f-f6f2-43bc-8e55-c4974ac964ec-kube-api-access-w992q\") pod \"maas-api-revoke-sub-56327c97-z4lt4\" (UID: \"2454f93f-f6f2-43bc-8e55-c4974ac964ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" Nov 03 22:02:29.373796 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.373758 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2454f93f-f6f2-43bc-8e55-c4974ac964ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56327c97-z4lt4\" (UID: \"2454f93f-f6f2-43bc-8e55-c4974ac964ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" Nov 03 22:02:29.373983 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.373868 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2454f93f-f6f2-43bc-8e55-c4974ac964ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56327c97-z4lt4\" (UID: \"2454f93f-f6f2-43bc-8e55-c4974ac964ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" Nov 03 22:02:29.373983 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.373925 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w992q\" (UniqueName: \"kubernetes.io/projected/2454f93f-f6f2-43bc-8e55-c4974ac964ec-kube-api-access-w992q\") pod \"maas-api-revoke-sub-56327c97-z4lt4\" (UID: \"2454f93f-f6f2-43bc-8e55-c4974ac964ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" Nov 03 22:02:29.374458 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.374434 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2454f93f-f6f2-43bc-8e55-c4974ac964ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56327c97-z4lt4\" (UID: \"2454f93f-f6f2-43bc-8e55-c4974ac964ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" Nov 03 22:02:29.376202 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.376178 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2454f93f-f6f2-43bc-8e55-c4974ac964ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56327c97-z4lt4\" (UID: \"2454f93f-f6f2-43bc-8e55-c4974ac964ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" Nov 03 22:02:29.394435 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.394407 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w992q\" (UniqueName: \"kubernetes.io/projected/2454f93f-f6f2-43bc-8e55-c4974ac964ec-kube-api-access-w992q\") pod \"maas-api-revoke-sub-56327c97-z4lt4\" (UID: \"2454f93f-f6f2-43bc-8e55-c4974ac964ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" Nov 03 22:02:29.495309 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.495277 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" Nov 03 22:02:29.788729 ip-10-0-131-253 kubenswrapper[5041]: W1103 22:02:29.788701 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2454f93f_f6f2_43bc_8e55_c4974ac964ec.slice/crio-59f43929c9635b2d4fb80cb081836d58328207a709480eada25bcfd4e1fecaed WatchSource:0}: Error finding container 59f43929c9635b2d4fb80cb081836d58328207a709480eada25bcfd4e1fecaed: Status 404 returned error can't find the container with id 59f43929c9635b2d4fb80cb081836d58328207a709480eada25bcfd4e1fecaed Nov 03 22:02:29.790138 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:29.790103 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4"] Nov 03 22:02:30.312735 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:30.312695 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" event={"ID":"2454f93f-f6f2-43bc-8e55-c4974ac964ec","Type":"ContainerStarted","Data":"3213f878f5da190d412764ea521c0ed0906aade69a25ac7d1fdfd02a9bfcb507"} Nov 03 22:02:30.313111 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:30.312758 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" event={"ID":"2454f93f-f6f2-43bc-8e55-c4974ac964ec","Type":"ContainerStarted","Data":"59f43929c9635b2d4fb80cb081836d58328207a709480eada25bcfd4e1fecaed"} Nov 03 22:02:30.354588 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:30.354540 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" podStartSLOduration=1.354526022 podStartE2EDuration="1.354526022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:30.352868299 +0000 UTC m=+1965.860096580" watchObservedRunningTime="2026-11-03 22:02:30.354526022 +0000 UTC m=+1965.861754330" Nov 03 22:02:31.320769 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:31.320730 5041 generic.go:356] "Generic (PLEG): container finished" podID="2454f93f-f6f2-43bc-8e55-c4974ac964ec" containerID="3213f878f5da190d412764ea521c0ed0906aade69a25ac7d1fdfd02a9bfcb507" exitCode=0 Nov 03 22:02:31.321219 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:31.320797 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" event={"ID":"2454f93f-f6f2-43bc-8e55-c4974ac964ec","Type":"ContainerDied","Data":"3213f878f5da190d412764ea521c0ed0906aade69a25ac7d1fdfd02a9bfcb507"} Nov 03 22:02:32.469578 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:32.469551 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" Nov 03 22:02:32.505948 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:32.505919 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2454f93f-f6f2-43bc-8e55-c4974ac964ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2454f93f-f6f2-43bc-8e55-c4974ac964ec-maas-api-xks-ca\") pod \"2454f93f-f6f2-43bc-8e55-c4974ac964ec\" (UID: \"2454f93f-f6f2-43bc-8e55-c4974ac964ec\") " Nov 03 22:02:32.506114 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:32.505999 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2454f93f-f6f2-43bc-8e55-c4974ac964ec-kube-api-access-w992q\" (UniqueName: \"kubernetes.io/projected/2454f93f-f6f2-43bc-8e55-c4974ac964ec-kube-api-access-w992q\") pod \"2454f93f-f6f2-43bc-8e55-c4974ac964ec\" (UID: \"2454f93f-f6f2-43bc-8e55-c4974ac964ec\") " Nov 03 22:02:32.506114 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:32.506068 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2454f93f-f6f2-43bc-8e55-c4974ac964ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2454f93f-f6f2-43bc-8e55-c4974ac964ec-maas-api-openshift-ca\") pod \"2454f93f-f6f2-43bc-8e55-c4974ac964ec\" (UID: \"2454f93f-f6f2-43bc-8e55-c4974ac964ec\") " Nov 03 22:02:32.506409 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:32.506387 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2454f93f-f6f2-43bc-8e55-c4974ac964ec-maas-api-openshift-ca" pod "2454f93f-f6f2-43bc-8e55-c4974ac964ec" (UID: "2454f93f-f6f2-43bc-8e55-c4974ac964ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:02:32.508297 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:32.508252 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2454f93f-f6f2-43bc-8e55-c4974ac964ec-maas-api-xks-ca" pod "2454f93f-f6f2-43bc-8e55-c4974ac964ec" (UID: "2454f93f-f6f2-43bc-8e55-c4974ac964ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:02:32.508606 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:32.508577 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2454f93f-f6f2-43bc-8e55-c4974ac964ec-kube-api-access-w992q" pod "2454f93f-f6f2-43bc-8e55-c4974ac964ec" (UID: "2454f93f-f6f2-43bc-8e55-c4974ac964ec"). InnerVolumeSpecName "kube-api-access-w992q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:02:32.607591 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:32.607513 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2454f93f-f6f2-43bc-8e55-c4974ac964ec-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:02:32.607591 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:32.607543 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2454f93f-f6f2-43bc-8e55-c4974ac964ec-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:02:32.607591 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:32.607553 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w992q\" (UniqueName: \"kubernetes.io/projected/2454f93f-f6f2-43bc-8e55-c4974ac964ec-kube-api-access-w992q\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:02:33.335343 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:33.335306 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" Nov 03 22:02:33.335343 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:33.335339 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4" event={"ID":"2454f93f-f6f2-43bc-8e55-c4974ac964ec","Type":"ContainerDied","Data":"59f43929c9635b2d4fb80cb081836d58328207a709480eada25bcfd4e1fecaed"} Nov 03 22:02:33.335591 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:33.335390 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59f43929c9635b2d4fb80cb081836d58328207a709480eada25bcfd4e1fecaed" Nov 03 22:02:40.781716 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:02:40.781676 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:02:49.500853 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:49.500817 5041 scope.go:122] "RemoveContainer" containerID="36affe12e68b805bc49b575568e422a8e88a97ecff6217367a0b71ed5ccd08e4" Nov 03 22:02:49.510919 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:49.510893 5041 scope.go:122] "RemoveContainer" containerID="e51082780cec778825af4ab20ca3d7a897aaebc1b35d4b36e4b5ffe85622eaaf" Nov 03 22:02:58.861133 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.861090 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf"] Nov 03 22:02:58.862204 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.861897 5041 cpu_manager.go:414] "containerMap: removing container" podUID="2454f93f-f6f2-43bc-8e55-c4974ac964ec" containerName="revoke-keys" Nov 03 22:02:58.862204 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.861922 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2454f93f-f6f2-43bc-8e55-c4974ac964ec" containerName="revoke-keys" Nov 03 22:02:58.862204 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.862084 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="2454f93f-f6f2-43bc-8e55-c4974ac964ec" containerName="revoke-keys" Nov 03 22:02:58.868209 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.868184 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.893515 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.893479 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf"] Nov 03 22:02:58.897518 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.897478 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6981249-e629-4de7-afdb-a34e4668704a-istio-podinfo\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.897747 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.897530 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-istio-envoy\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.897747 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.897627 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-workload-certs\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.897903 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.897774 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-credential-socket\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.897903 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.897813 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-workload-socket\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.898022 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.897916 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6981249-e629-4de7-afdb-a34e4668704a-istio-token\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.898022 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.897987 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-istio-data\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.898130 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.898043 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g8wk\" (UniqueName: \"kubernetes.io/projected/d6981249-e629-4de7-afdb-a34e4668704a-kube-api-access-9g8wk\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.898130 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.898081 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6981249-e629-4de7-afdb-a34e4668704a-istiod-ca-cert\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.901939 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.901891 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b14ec9a0-openshift-default-dockercfg-ngs4s\"" Nov 03 22:02:58.998828 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.998788 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-istio-data\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.998828 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.998830 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g8wk\" (UniqueName: \"kubernetes.io/projected/d6981249-e629-4de7-afdb-a34e4668704a-kube-api-access-9g8wk\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.999087 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.998850 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6981249-e629-4de7-afdb-a34e4668704a-istiod-ca-cert\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.999087 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.998878 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6981249-e629-4de7-afdb-a34e4668704a-istio-podinfo\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.999087 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.998899 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-istio-envoy\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.999087 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.998916 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-workload-certs\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.999087 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.998991 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-credential-socket\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.999087 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.999025 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-workload-socket\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.999087 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.999071 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6981249-e629-4de7-afdb-a34e4668704a-istio-token\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.999480 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.999303 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-istio-data\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.999480 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.999444 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-credential-socket\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.999602 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.999545 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-workload-socket\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.999749 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.999727 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6981249-e629-4de7-afdb-a34e4668704a-istiod-ca-cert\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:58.999824 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:58.999759 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-workload-certs\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:59.001408 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:59.001376 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-istio-envoy\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:59.001583 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:59.001563 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6981249-e629-4de7-afdb-a34e4668704a-istio-podinfo\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:59.019191 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:59.019162 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6981249-e629-4de7-afdb-a34e4668704a-istio-token\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:59.019309 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:59.019259 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g8wk\" (UniqueName: \"kubernetes.io/projected/d6981249-e629-4de7-afdb-a34e4668704a-kube-api-access-9g8wk\") pod \"e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:59.185555 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:59.185457 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:02:59.503858 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:59.503825 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf"] Nov 03 22:02:59.504410 ip-10-0-131-253 kubenswrapper[5041]: W1103 22:02:59.504382 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6981249_e629_4de7_afdb_a34e4668704a.slice/crio-5dfb073349bd3cde24201c48563e6dfeb4e3e48364634394273f0c89ca787701 WatchSource:0}: Error finding container 5dfb073349bd3cde24201c48563e6dfeb4e3e48364634394273f0c89ca787701: Status 404 returned error can't find the container with id 5dfb073349bd3cde24201c48563e6dfeb4e3e48364634394273f0c89ca787701 Nov 03 22:02:59.506653 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:59.506625 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:02:59.506741 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:59.506703 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:02:59.506741 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:59.506736 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:02:59.515503 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:02:59.515470 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" event={"ID":"d6981249-e629-4de7-afdb-a34e4668704a","Type":"ContainerStarted","Data":"5dfb073349bd3cde24201c48563e6dfeb4e3e48364634394273f0c89ca787701"} Nov 03 22:03:00.522834 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:00.522751 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" event={"ID":"d6981249-e629-4de7-afdb-a34e4668704a","Type":"ContainerStarted","Data":"8d2de54964dbbf1c38a16cd5d846e46b0f0a261aa6602bd41de250c327b3f33a"} Nov 03 22:03:00.583102 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:00.583037 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" podStartSLOduration=2.58302132 podStartE2EDuration="2.58302132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:03:00.579645666 +0000 UTC m=+1996.086873947" watchObservedRunningTime="2026-11-03 22:03:00.58302132 +0000 UTC m=+1996.090249600" Nov 03 22:03:01.185916 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:01.185881 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:03:01.191242 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:01.191211 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:03:01.528998 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:01.528954 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:03:01.529875 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:01.529853 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:03:10.777884 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:03:10.777842 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:03:17.569841 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:17.569798 5041 patch_prober.go:29] interesting pod/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:03:17.570254 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:17.569902 5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" podUID="d6981249-e629-4de7-afdb-a34e4668704a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:03:17.996361 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:17.996320 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66"] Nov 03 22:03:17.999258 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:17.999235 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" Nov 03 22:03:18.020217 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.020186 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66"] Nov 03 22:03:18.030074 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.030049 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:03:18.030074 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.030070 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p4zs2\"" Nov 03 22:03:18.129951 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.129914 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8450826-f349-40a7-ae33-1ef00c5eabdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66\" (UID: \"f8450826-f349-40a7-ae33-1ef00c5eabdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" Nov 03 22:03:18.130136 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.129971 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8450826-f349-40a7-ae33-1ef00c5eabdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66\" (UID: \"f8450826-f349-40a7-ae33-1ef00c5eabdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" Nov 03 22:03:18.130136 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.130093 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djwnx\" (UniqueName: \"kubernetes.io/projected/f8450826-f349-40a7-ae33-1ef00c5eabdf-kube-api-access-djwnx\") pod \"maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66\" (UID: \"f8450826-f349-40a7-ae33-1ef00c5eabdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" Nov 03 22:03:18.231510 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.231473 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8450826-f349-40a7-ae33-1ef00c5eabdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66\" (UID: \"f8450826-f349-40a7-ae33-1ef00c5eabdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" Nov 03 22:03:18.231757 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.231539 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8450826-f349-40a7-ae33-1ef00c5eabdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66\" (UID: \"f8450826-f349-40a7-ae33-1ef00c5eabdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" Nov 03 22:03:18.231757 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.231650 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djwnx\" (UniqueName: \"kubernetes.io/projected/f8450826-f349-40a7-ae33-1ef00c5eabdf-kube-api-access-djwnx\") pod \"maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66\" (UID: \"f8450826-f349-40a7-ae33-1ef00c5eabdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" Nov 03 22:03:18.232158 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.232137 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8450826-f349-40a7-ae33-1ef00c5eabdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66\" (UID: \"f8450826-f349-40a7-ae33-1ef00c5eabdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" Nov 03 22:03:18.234081 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.234055 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8450826-f349-40a7-ae33-1ef00c5eabdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66\" (UID: \"f8450826-f349-40a7-ae33-1ef00c5eabdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" Nov 03 22:03:18.262427 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:03:18.262351 5041 projected.go:196] Error preparing data for projected volume kube-api-access-djwnx for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 03 22:03:18.262427 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:03:18.262421 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8450826-f349-40a7-ae33-1ef00c5eabdf-kube-api-access-djwnx podName:f8450826-f349-40a7-ae33-1ef00c5eabdf nodeName:}" failed. No retries permitted until 2026-11-03 22:03:18.762403396 +0000 UTC m=+2014.269631659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-djwnx" (UniqueName: "kubernetes.io/projected/f8450826-f349-40a7-ae33-1ef00c5eabdf-kube-api-access-djwnx") pod "maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" (UID: "f8450826-f349-40a7-ae33-1ef00c5eabdf") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 03 22:03:18.667938 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.667847 5041 generic.go:356] "Generic (PLEG): container finished" podID="d6981249-e629-4de7-afdb-a34e4668704a" containerID="8d2de54964dbbf1c38a16cd5d846e46b0f0a261aa6602bd41de250c327b3f33a" exitCode=0 Nov 03 22:03:18.668328 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.667930 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" event={"ID":"d6981249-e629-4de7-afdb-a34e4668704a","Type":"ContainerDied","Data":"8d2de54964dbbf1c38a16cd5d846e46b0f0a261aa6602bd41de250c327b3f33a"} Nov 03 22:03:18.668457 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.668439 5041 scope.go:122] "RemoveContainer" containerID="8d2de54964dbbf1c38a16cd5d846e46b0f0a261aa6602bd41de250c327b3f33a" Nov 03 22:03:18.669846 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.669817 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:03:18.669942 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.669886 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:03:18.669942 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.669919 5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:03:18.839513 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:18.839468 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djwnx\" (UniqueName: \"kubernetes.io/projected/f8450826-f349-40a7-ae33-1ef00c5eabdf-kube-api-access-djwnx\") pod \"maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66\" (UID: \"f8450826-f349-40a7-ae33-1ef00c5eabdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" Nov 03 22:03:18.855415 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:03:18.855384 5041 projected.go:196] Error preparing data for projected volume kube-api-access-djwnx for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 03 22:03:18.855552 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:03:18.855454 5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8450826-f349-40a7-ae33-1ef00c5eabdf-kube-api-access-djwnx podName:f8450826-f349-40a7-ae33-1ef00c5eabdf nodeName:}" failed. No retries permitted until 2026-11-03 22:03:19.855438409 +0000 UTC m=+2015.362666680 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-djwnx" (UniqueName: "kubernetes.io/projected/f8450826-f349-40a7-ae33-1ef00c5eabdf-kube-api-access-djwnx") pod "maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" (UID: "f8450826-f349-40a7-ae33-1ef00c5eabdf") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 03 22:03:19.676370 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:19.676331 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" event={"ID":"d6981249-e629-4de7-afdb-a34e4668704a","Type":"ContainerStarted","Data":"1aafe62ea91c73283a0253727ed1727d2e85d91a20b4bdbee3f3033923604878"} Nov 03 22:03:19.951050 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:19.950984 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djwnx\" (UniqueName: \"kubernetes.io/projected/f8450826-f349-40a7-ae33-1ef00c5eabdf-kube-api-access-djwnx\") pod \"maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66\" (UID: \"f8450826-f349-40a7-ae33-1ef00c5eabdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" Nov 03 22:03:19.971798 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:19.971764 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djwnx\" (UniqueName: \"kubernetes.io/projected/f8450826-f349-40a7-ae33-1ef00c5eabdf-kube-api-access-djwnx\") pod \"maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66\" (UID: \"f8450826-f349-40a7-ae33-1ef00c5eabdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" Nov 03 22:03:20.123739 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:20.122870 5041 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" secret="" err="secret \"maas-api-cleanup-dockercfg-p4zs2\" not found" Nov 03 22:03:20.123739 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:20.122947 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" Nov 03 22:03:20.186984 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:20.186952 5041 patch_prober.go:29] interesting pod/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 22:03:20.187105 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:20.187015 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" podUID="d6981249-e629-4de7-afdb-a34e4668704a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 22:03:20.187337 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:20.186529 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:03:20.437053 ip-10-0-131-253 kubenswrapper[5041]: W1103 22:03:20.437017 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8450826_f349_40a7_ae33_1ef00c5eabdf.slice/crio-d814e155116e6ac9d46bf4f84d068c0316d1f81d8ae04046e7243e3d7a2491f2 WatchSource:0}: Error finding container d814e155116e6ac9d46bf4f84d068c0316d1f81d8ae04046e7243e3d7a2491f2: Status 404 returned error can't find the container with id d814e155116e6ac9d46bf4f84d068c0316d1f81d8ae04046e7243e3d7a2491f2 Nov 03 22:03:20.437511 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:20.437487 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66"] Nov 03 22:03:20.684519 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:20.684418 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" event={"ID":"f8450826-f349-40a7-ae33-1ef00c5eabdf","Type":"ContainerStarted","Data":"d814e155116e6ac9d46bf4f84d068c0316d1f81d8ae04046e7243e3d7a2491f2"} Nov 03 22:03:21.186043 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:21.186003 5041 patch_prober.go:29] interesting pod/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 22:03:21.186240 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:21.186070 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" podUID="d6981249-e629-4de7-afdb-a34e4668704a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 22:03:21.692693 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:21.692594 5041 generic.go:356] "Generic (PLEG): container finished" podID="f8450826-f349-40a7-ae33-1ef00c5eabdf" containerID="94376b313ec4f8c324b44f764a351339d1aa2a245d4b47fde4db7d41782230af" exitCode=0 Nov 03 22:03:21.692693 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:21.692658 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" event={"ID":"f8450826-f349-40a7-ae33-1ef00c5eabdf","Type":"ContainerDied","Data":"94376b313ec4f8c324b44f764a351339d1aa2a245d4b47fde4db7d41782230af"} Nov 03 22:03:22.186438 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:22.186390 5041 patch_prober.go:29] interesting pod/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 22:03:22.186623 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:22.186460 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" podUID="d6981249-e629-4de7-afdb-a34e4668704a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 22:03:22.940271 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:22.940238 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" Nov 03 22:03:23.085076 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:23.085039 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8450826-f349-40a7-ae33-1ef00c5eabdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8450826-f349-40a7-ae33-1ef00c5eabdf-maas-api-openshift-ca\") pod \"f8450826-f349-40a7-ae33-1ef00c5eabdf\" (UID: \"f8450826-f349-40a7-ae33-1ef00c5eabdf\") " Nov 03 22:03:23.085263 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:23.085115 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8450826-f349-40a7-ae33-1ef00c5eabdf-kube-api-access-djwnx\" (UniqueName: \"kubernetes.io/projected/f8450826-f349-40a7-ae33-1ef00c5eabdf-kube-api-access-djwnx\") pod \"f8450826-f349-40a7-ae33-1ef00c5eabdf\" (UID: \"f8450826-f349-40a7-ae33-1ef00c5eabdf\") " Nov 03 22:03:23.085263 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:23.085238 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8450826-f349-40a7-ae33-1ef00c5eabdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8450826-f349-40a7-ae33-1ef00c5eabdf-maas-api-xks-ca\") pod \"f8450826-f349-40a7-ae33-1ef00c5eabdf\" (UID: \"f8450826-f349-40a7-ae33-1ef00c5eabdf\") " Nov 03 22:03:23.085359 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:23.085336 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8450826-f349-40a7-ae33-1ef00c5eabdf-maas-api-openshift-ca" pod "f8450826-f349-40a7-ae33-1ef00c5eabdf" (UID: "f8450826-f349-40a7-ae33-1ef00c5eabdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:03:23.085571 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:23.085551 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8450826-f349-40a7-ae33-1ef00c5eabdf-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:03:23.087440 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:23.087416 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8450826-f349-40a7-ae33-1ef00c5eabdf-maas-api-xks-ca" pod "f8450826-f349-40a7-ae33-1ef00c5eabdf" (UID: "f8450826-f349-40a7-ae33-1ef00c5eabdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:03:23.087523 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:23.087507 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8450826-f349-40a7-ae33-1ef00c5eabdf-kube-api-access-djwnx" pod "f8450826-f349-40a7-ae33-1ef00c5eabdf" (UID: "f8450826-f349-40a7-ae33-1ef00c5eabdf"). InnerVolumeSpecName "kube-api-access-djwnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:23.185904 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:23.185869 5041 patch_prober.go:29] interesting pod/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 22:03:23.186091 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:23.185918 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" podUID="d6981249-e629-4de7-afdb-a34e4668704a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 22:03:23.186223 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:23.186201 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djwnx\" (UniqueName: \"kubernetes.io/projected/f8450826-f349-40a7-ae33-1ef00c5eabdf-kube-api-access-djwnx\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:03:23.186265 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:23.186228 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8450826-f349-40a7-ae33-1ef00c5eabdf-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:03:23.706434 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:23.706398 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" event={"ID":"f8450826-f349-40a7-ae33-1ef00c5eabdf","Type":"ContainerDied","Data":"d814e155116e6ac9d46bf4f84d068c0316d1f81d8ae04046e7243e3d7a2491f2"} Nov 03 22:03:23.706434 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:23.706418 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66" Nov 03 22:03:23.706653 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:23.706449 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d814e155116e6ac9d46bf4f84d068c0316d1f81d8ae04046e7243e3d7a2491f2" Nov 03 22:03:24.186855 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:24.186765 5041 patch_prober.go:29] interesting pod/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 22:03:24.186855 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:24.186838 5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" podUID="d6981249-e629-4de7-afdb-a34e4668704a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 22:03:25.189500 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:25.189465 5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:03:25.189941 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:25.189915 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:03:25.190541 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:25.190521 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:03:30.089955 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:30.089744 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66"] Nov 03 22:03:30.100191 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:30.100155 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b14ec9a0-rlt66"] Nov 03 22:03:30.629091 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:30.629042 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8450826-f349-40a7-ae33-1ef00c5eabdf" path="/var/lib/kubelet/pods/f8450826-f349-40a7-ae33-1ef00c5eabdf/volumes" Nov 03 22:03:34.239591 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:34.239552 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf"] Nov 03 22:03:34.240107 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:34.239845 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" podUID="d6981249-e629-4de7-afdb-a34e4668704a" containerName="istio-proxy" containerID="cri-o://1aafe62ea91c73283a0253727ed1727d2e85d91a20b4bdbee3f3033923604878" gracePeriod=30 Nov 03 22:03:39.508819 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.508750 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:03:39.579703 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.579603 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6981249-e629-4de7-afdb-a34e4668704a-kube-api-access-9g8wk\" (UniqueName: \"kubernetes.io/projected/d6981249-e629-4de7-afdb-a34e4668704a-kube-api-access-9g8wk\") pod \"d6981249-e629-4de7-afdb-a34e4668704a\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " Nov 03 22:03:39.579936 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.579769 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-workload-certs\") pod \"d6981249-e629-4de7-afdb-a34e4668704a\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " Nov 03 22:03:39.579936 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.579814 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-credential-socket\") pod \"d6981249-e629-4de7-afdb-a34e4668704a\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " Nov 03 22:03:39.579936 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.579847 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6981249-e629-4de7-afdb-a34e4668704a-istio-token\" (UniqueName: \"kubernetes.io/projected/d6981249-e629-4de7-afdb-a34e4668704a-istio-token\") pod \"d6981249-e629-4de7-afdb-a34e4668704a\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " Nov 03 22:03:39.580120 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.580015 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-workload-socket\") pod \"d6981249-e629-4de7-afdb-a34e4668704a\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " Nov 03 22:03:39.580181 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.580121 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6981249-e629-4de7-afdb-a34e4668704a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6981249-e629-4de7-afdb-a34e4668704a-istiod-ca-cert\") pod \"d6981249-e629-4de7-afdb-a34e4668704a\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " Nov 03 22:03:39.580181 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.580147 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-credential-socket" pod "d6981249-e629-4de7-afdb-a34e4668704a" (UID: "d6981249-e629-4de7-afdb-a34e4668704a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:39.580181 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.580164 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-istio-data\") pod \"d6981249-e629-4de7-afdb-a34e4668704a\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " Nov 03 22:03:39.580181 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.580169 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-workload-certs" pod "d6981249-e629-4de7-afdb-a34e4668704a" (UID: "d6981249-e629-4de7-afdb-a34e4668704a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:39.580181 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.580192 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-istio-envoy\") pod \"d6981249-e629-4de7-afdb-a34e4668704a\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " Nov 03 22:03:39.580181 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.580246 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6981249-e629-4de7-afdb-a34e4668704a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6981249-e629-4de7-afdb-a34e4668704a-istio-podinfo\") pod \"d6981249-e629-4de7-afdb-a34e4668704a\" (UID: \"d6981249-e629-4de7-afdb-a34e4668704a\") " Nov 03 22:03:39.580990 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.580368 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-workload-socket" pod "d6981249-e629-4de7-afdb-a34e4668704a" (UID: "d6981249-e629-4de7-afdb-a34e4668704a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:39.580990 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.580490 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6981249-e629-4de7-afdb-a34e4668704a-istiod-ca-cert" pod "d6981249-e629-4de7-afdb-a34e4668704a" (UID: "d6981249-e629-4de7-afdb-a34e4668704a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:03:39.580990 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.580756 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-istio-data" pod "d6981249-e629-4de7-afdb-a34e4668704a" (UID: "d6981249-e629-4de7-afdb-a34e4668704a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:39.581184 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.580995 5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-workload-certs\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:03:39.581184 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.581029 5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-credential-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:03:39.581184 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.581045 5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-workload-socket\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:03:39.581184 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.581058 5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6981249-e629-4de7-afdb-a34e4668704a-istiod-ca-cert\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:03:39.581184 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.581071 5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-istio-data\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:03:39.582536 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.582509 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6981249-e629-4de7-afdb-a34e4668704a-kube-api-access-9g8wk" pod "d6981249-e629-4de7-afdb-a34e4668704a" (UID: "d6981249-e629-4de7-afdb-a34e4668704a"). InnerVolumeSpecName "kube-api-access-9g8wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:39.582963 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.582934 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6981249-e629-4de7-afdb-a34e4668704a-istio-podinfo" pod "d6981249-e629-4de7-afdb-a34e4668704a" (UID: "d6981249-e629-4de7-afdb-a34e4668704a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:03:39.583127 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.583110 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6981249-e629-4de7-afdb-a34e4668704a-istio-token" pod "d6981249-e629-4de7-afdb-a34e4668704a" (UID: "d6981249-e629-4de7-afdb-a34e4668704a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:03:39.583193 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.583153 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-istio-envoy" pod "d6981249-e629-4de7-afdb-a34e4668704a" (UID: "d6981249-e629-4de7-afdb-a34e4668704a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:03:39.682547 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.682506 5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6981249-e629-4de7-afdb-a34e4668704a-istio-envoy\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:03:39.682547 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.682537 5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6981249-e629-4de7-afdb-a34e4668704a-istio-podinfo\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:03:39.682547 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.682546 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g8wk\" (UniqueName: \"kubernetes.io/projected/d6981249-e629-4de7-afdb-a34e4668704a-kube-api-access-9g8wk\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:03:39.682547 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.682557 5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6981249-e629-4de7-afdb-a34e4668704a-istio-token\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:03:39.814547 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.814452 5041 generic.go:356] "Generic (PLEG): container finished" podID="d6981249-e629-4de7-afdb-a34e4668704a" containerID="1aafe62ea91c73283a0253727ed1727d2e85d91a20b4bdbee3f3033923604878" exitCode=0 Nov 03 22:03:39.814547 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.814518 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" Nov 03 22:03:39.814547 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.814530 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" event={"ID":"d6981249-e629-4de7-afdb-a34e4668704a","Type":"ContainerDied","Data":"1aafe62ea91c73283a0253727ed1727d2e85d91a20b4bdbee3f3033923604878"} Nov 03 22:03:39.814880 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.814596 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf" event={"ID":"d6981249-e629-4de7-afdb-a34e4668704a","Type":"ContainerDied","Data":"5dfb073349bd3cde24201c48563e6dfeb4e3e48364634394273f0c89ca787701"} Nov 03 22:03:39.814880 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.814624 5041 scope.go:122] "RemoveContainer" containerID="1aafe62ea91c73283a0253727ed1727d2e85d91a20b4bdbee3f3033923604878" Nov 03 22:03:39.828840 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.828815 5041 scope.go:122] "RemoveContainer" containerID="8d2de54964dbbf1c38a16cd5d846e46b0f0a261aa6602bd41de250c327b3f33a" Nov 03 22:03:39.839228 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.839201 5041 scope.go:122] "RemoveContainer" containerID="1aafe62ea91c73283a0253727ed1727d2e85d91a20b4bdbee3f3033923604878" Nov 03 22:03:39.839523 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:03:39.839500 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aafe62ea91c73283a0253727ed1727d2e85d91a20b4bdbee3f3033923604878\": container with ID starting with 1aafe62ea91c73283a0253727ed1727d2e85d91a20b4bdbee3f3033923604878 not found: ID does not exist" containerID="1aafe62ea91c73283a0253727ed1727d2e85d91a20b4bdbee3f3033923604878" Nov 03 22:03:39.839523 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.839543 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aafe62ea91c73283a0253727ed1727d2e85d91a20b4bdbee3f3033923604878"} err="failed to get container status \"1aafe62ea91c73283a0253727ed1727d2e85d91a20b4bdbee3f3033923604878\": rpc error: code = NotFound desc = could not find container \"1aafe62ea91c73283a0253727ed1727d2e85d91a20b4bdbee3f3033923604878\": container with ID starting with 1aafe62ea91c73283a0253727ed1727d2e85d91a20b4bdbee3f3033923604878 not found: ID does not exist" Nov 03 22:03:39.839736 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.839563 5041 scope.go:122] "RemoveContainer" containerID="8d2de54964dbbf1c38a16cd5d846e46b0f0a261aa6602bd41de250c327b3f33a" Nov 03 22:03:39.839894 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:03:39.839875 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d2de54964dbbf1c38a16cd5d846e46b0f0a261aa6602bd41de250c327b3f33a\": container with ID starting with 8d2de54964dbbf1c38a16cd5d846e46b0f0a261aa6602bd41de250c327b3f33a not found: ID does not exist" containerID="8d2de54964dbbf1c38a16cd5d846e46b0f0a261aa6602bd41de250c327b3f33a" Nov 03 22:03:39.839959 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.839898 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d2de54964dbbf1c38a16cd5d846e46b0f0a261aa6602bd41de250c327b3f33a"} err="failed to get container status \"8d2de54964dbbf1c38a16cd5d846e46b0f0a261aa6602bd41de250c327b3f33a\": rpc error: code = NotFound desc = could not find container \"8d2de54964dbbf1c38a16cd5d846e46b0f0a261aa6602bd41de250c327b3f33a\": container with ID starting with 8d2de54964dbbf1c38a16cd5d846e46b0f0a261aa6602bd41de250c327b3f33a not found: ID does not exist" Nov 03 22:03:39.876871 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.876841 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf"] Nov 03 22:03:39.880850 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:39.880826 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b14ec9a0-openshift-default-6556d644f5-zc2pf"] Nov 03 22:03:40.620943 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:40.620909 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6981249-e629-4de7-afdb-a34e4668704a" path="/var/lib/kubelet/pods/d6981249-e629-4de7-afdb-a34e4668704a/volumes" Nov 03 22:03:40.777803 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:03:40.777766 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:03:41.226536 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.226504 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv"] Nov 03 22:03:41.227029 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.227013 5041 cpu_manager.go:414] "containerMap: removing container" podUID="d6981249-e629-4de7-afdb-a34e4668704a" containerName="istio-proxy" Nov 03 22:03:41.227029 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.227029 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6981249-e629-4de7-afdb-a34e4668704a" containerName="istio-proxy" Nov 03 22:03:41.227125 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.227041 5041 cpu_manager.go:414] "containerMap: removing container" podUID="d6981249-e629-4de7-afdb-a34e4668704a" containerName="istio-proxy" Nov 03 22:03:41.227125 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.227047 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6981249-e629-4de7-afdb-a34e4668704a" containerName="istio-proxy" Nov 03 22:03:41.227125 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.227078 5041 cpu_manager.go:414] "containerMap: removing container" podUID="f8450826-f349-40a7-ae33-1ef00c5eabdf" containerName="revoke-keys" Nov 03 22:03:41.227125 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.227084 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8450826-f349-40a7-ae33-1ef00c5eabdf" containerName="revoke-keys" Nov 03 22:03:41.227255 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.227162 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8450826-f349-40a7-ae33-1ef00c5eabdf" containerName="revoke-keys" Nov 03 22:03:41.227255 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.227174 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6981249-e629-4de7-afdb-a34e4668704a" containerName="istio-proxy" Nov 03 22:03:41.227255 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.227180 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6981249-e629-4de7-afdb-a34e4668704a" containerName="istio-proxy" Nov 03 22:03:41.230544 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.230527 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" Nov 03 22:03:41.250717 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.248881 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv"] Nov 03 22:03:41.252703 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.252674 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n"] Nov 03 22:03:41.255637 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.255619 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" Nov 03 22:03:41.258379 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.258354 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-6a697e\"" Nov 03 22:03:41.263482 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.263463 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-6a697e-dockercfg-622dr\"" Nov 03 22:03:41.281323 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.281294 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n"] Nov 03 22:03:41.304946 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.304912 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bccca265-6537-43d2-b5c1-d301283c97a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv\" (UID: \"bccca265-6537-43d2-b5c1-d301283c97a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" Nov 03 22:03:41.305077 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.304973 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkkdw\" (UniqueName: \"kubernetes.io/projected/bccca265-6537-43d2-b5c1-d301283c97a2-kube-api-access-mkkdw\") pod \"payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv\" (UID: \"bccca265-6537-43d2-b5c1-d301283c97a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" Nov 03 22:03:41.305120 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.305081 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107e2265-6272-44e7-9ae8-eb86c7038b71-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n\" (UID: \"107e2265-6272-44e7-9ae8-eb86c7038b71\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" Nov 03 22:03:41.305244 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.305225 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6lck\" (UniqueName: \"kubernetes.io/projected/107e2265-6272-44e7-9ae8-eb86c7038b71-kube-api-access-b6lck\") pod \"payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n\" (UID: \"107e2265-6272-44e7-9ae8-eb86c7038b71\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" Nov 03 22:03:41.405818 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.405781 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bccca265-6537-43d2-b5c1-d301283c97a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv\" (UID: \"bccca265-6537-43d2-b5c1-d301283c97a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" Nov 03 22:03:41.406025 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.405842 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkkdw\" (UniqueName: \"kubernetes.io/projected/bccca265-6537-43d2-b5c1-d301283c97a2-kube-api-access-mkkdw\") pod \"payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv\" (UID: \"bccca265-6537-43d2-b5c1-d301283c97a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" Nov 03 22:03:41.406025 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.405951 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107e2265-6272-44e7-9ae8-eb86c7038b71-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n\" (UID: \"107e2265-6272-44e7-9ae8-eb86c7038b71\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" Nov 03 22:03:41.406160 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.406067 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6lck\" (UniqueName: \"kubernetes.io/projected/107e2265-6272-44e7-9ae8-eb86c7038b71-kube-api-access-b6lck\") pod \"payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n\" (UID: \"107e2265-6272-44e7-9ae8-eb86c7038b71\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" Nov 03 22:03:41.406599 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.406574 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bccca265-6537-43d2-b5c1-d301283c97a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv\" (UID: \"bccca265-6537-43d2-b5c1-d301283c97a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" Nov 03 22:03:41.406719 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.406654 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107e2265-6272-44e7-9ae8-eb86c7038b71-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n\" (UID: \"107e2265-6272-44e7-9ae8-eb86c7038b71\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" Nov 03 22:03:41.426223 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.426197 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6lck\" (UniqueName: \"kubernetes.io/projected/107e2265-6272-44e7-9ae8-eb86c7038b71-kube-api-access-b6lck\") pod \"payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n\" (UID: \"107e2265-6272-44e7-9ae8-eb86c7038b71\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" Nov 03 22:03:41.426361 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.426257 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkkdw\" (UniqueName: \"kubernetes.io/projected/bccca265-6537-43d2-b5c1-d301283c97a2-kube-api-access-mkkdw\") pod \"payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv\" (UID: \"bccca265-6537-43d2-b5c1-d301283c97a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" Nov 03 22:03:41.543481 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.543399 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" Nov 03 22:03:41.569270 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.569241 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" Nov 03 22:03:41.842026 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.841996 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv"] Nov 03 22:03:41.842413 ip-10-0-131-253 kubenswrapper[5041]: W1103 22:03:41.842054 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbccca265_6537_43d2_b5c1_d301283c97a2.slice/crio-e9361fa0113159a257754803d3f84f9834a5943392bee7dc37545f94ebe3a439 WatchSource:0}: Error finding container e9361fa0113159a257754803d3f84f9834a5943392bee7dc37545f94ebe3a439: Status 404 returned error can't find the container with id e9361fa0113159a257754803d3f84f9834a5943392bee7dc37545f94ebe3a439 Nov 03 22:03:41.877119 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:41.877091 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n"] Nov 03 22:03:41.885113 ip-10-0-131-253 kubenswrapper[5041]: W1103 22:03:41.885082 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107e2265_6272_44e7_9ae8_eb86c7038b71.slice/crio-96fa0d799c68c686b3e9d2b1c96d591b1bef27ab5f4c25c840df2d88a08c4ffd WatchSource:0}: Error finding container 96fa0d799c68c686b3e9d2b1c96d591b1bef27ab5f4c25c840df2d88a08c4ffd: Status 404 returned error can't find the container with id 96fa0d799c68c686b3e9d2b1c96d591b1bef27ab5f4c25c840df2d88a08c4ffd Nov 03 22:03:42.836486 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:42.836445 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" event={"ID":"bccca265-6537-43d2-b5c1-d301283c97a2","Type":"ContainerStarted","Data":"a4e641aba60fdb44c19b65e595878c8ad3082612a5b99719bf72e6a5d3b99fc5"} Nov 03 22:03:42.836729 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:42.836507 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" event={"ID":"bccca265-6537-43d2-b5c1-d301283c97a2","Type":"ContainerStarted","Data":"e9361fa0113159a257754803d3f84f9834a5943392bee7dc37545f94ebe3a439"} Nov 03 22:03:42.836729 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:42.836682 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" Nov 03 22:03:42.838430 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:42.838406 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" event={"ID":"107e2265-6272-44e7-9ae8-eb86c7038b71","Type":"ContainerStarted","Data":"755961cb4d2079bc261118fefa5dad732fd8fe8f59f749bce08fde1b998a2b99"} Nov 03 22:03:42.838560 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:42.838454 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" Nov 03 22:03:42.838560 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:42.838478 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" event={"ID":"107e2265-6272-44e7-9ae8-eb86c7038b71","Type":"ContainerStarted","Data":"96fa0d799c68c686b3e9d2b1c96d591b1bef27ab5f4c25c840df2d88a08c4ffd"} Nov 03 22:03:42.878584 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:42.878533 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" podStartSLOduration=1.8785188050000001 podStartE2EDuration="1.878518805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:03:42.877229747 +0000 UTC m=+2038.384458047" watchObservedRunningTime="2026-11-03 22:03:42.878518805 +0000 UTC m=+2038.385747085" Nov 03 22:03:42.920650 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:42.920584 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" podStartSLOduration=1.920567929 podStartE2EDuration="1.920567929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:03:42.917088125 +0000 UTC m=+2038.424316429" watchObservedRunningTime="2026-11-03 22:03:42.920567929 +0000 UTC m=+2038.427796209" Nov 03 22:03:49.543955 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:49.543923 5041 scope.go:122] "RemoveContainer" containerID="fa2cf4058465f21f71a8687c6c0ad03e4edbbe17697394db75a06d106df77ca8" Nov 03 22:03:53.849010 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:53.848975 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" Nov 03 22:03:53.849410 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:53.849197 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" Nov 03 22:03:55.028737 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:55.028695 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb"] Nov 03 22:03:55.040742 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:55.040710 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7szb"] Nov 03 22:03:56.627212 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:03:56.627175 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07fdee8c-dfe3-4ada-9999-c0e073b7187d" path="/var/lib/kubelet/pods/07fdee8c-dfe3-4ada-9999-c0e073b7187d/volumes" Nov 03 22:04:10.777542 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:04:10.777503 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:04:16.026858 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:04:16.026815 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq"] Nov 03 22:04:16.040760 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:04:16.040721 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxcmq"] Nov 03 22:04:16.621739 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:04:16.621698 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db24a304-683e-4536-863c-6d13c244fde3" path="/var/lib/kubelet/pods/db24a304-683e-4536-863c-6d13c244fde3/volumes" Nov 03 22:04:40.778087 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:04:40.778049 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:04:45.817532 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:04:45.817495 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 22:04:45.821961 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:04:45.821934 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 22:05:10.777475 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:05:10.777394 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:05:14.495976 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:14.495939 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t"] Nov 03 22:05:14.499249 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:14.499232 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t" Nov 03 22:05:14.516713 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:14.516691 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t"] Nov 03 22:05:14.530510 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:14.530491 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:05:14.530621 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:14.530568 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n4fdx\"" Nov 03 22:05:14.571799 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:14.571767 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11448cf2-8179-47e8-b5c6-b13139d0cac3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t\" (UID: \"11448cf2-8179-47e8-b5c6-b13139d0cac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t" Nov 03 22:05:14.571894 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:14.571804 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzndr\" (UniqueName: \"kubernetes.io/projected/11448cf2-8179-47e8-b5c6-b13139d0cac3-kube-api-access-qzndr\") pod \"maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t\" (UID: \"11448cf2-8179-47e8-b5c6-b13139d0cac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t" Nov 03 22:05:14.571894 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:14.571829 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11448cf2-8179-47e8-b5c6-b13139d0cac3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t\" (UID: \"11448cf2-8179-47e8-b5c6-b13139d0cac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t" Nov 03 22:05:14.672744 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:14.672714 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11448cf2-8179-47e8-b5c6-b13139d0cac3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t\" (UID: \"11448cf2-8179-47e8-b5c6-b13139d0cac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t" Nov 03 22:05:14.672943 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:14.672751 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzndr\" (UniqueName: \"kubernetes.io/projected/11448cf2-8179-47e8-b5c6-b13139d0cac3-kube-api-access-qzndr\") pod \"maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t\" (UID: \"11448cf2-8179-47e8-b5c6-b13139d0cac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t" Nov 03 22:05:14.672943 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:14.672776 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11448cf2-8179-47e8-b5c6-b13139d0cac3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t\" (UID: \"11448cf2-8179-47e8-b5c6-b13139d0cac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t" Nov 03 22:05:14.673554 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:14.673526 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11448cf2-8179-47e8-b5c6-b13139d0cac3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t\" (UID: \"11448cf2-8179-47e8-b5c6-b13139d0cac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t" Nov 03 22:05:14.675233 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:14.675210 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11448cf2-8179-47e8-b5c6-b13139d0cac3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t\" (UID: \"11448cf2-8179-47e8-b5c6-b13139d0cac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t" Nov 03 22:05:14.693153 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:14.693131 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzndr\" (UniqueName: \"kubernetes.io/projected/11448cf2-8179-47e8-b5c6-b13139d0cac3-kube-api-access-qzndr\") pod \"maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t\" (UID: \"11448cf2-8179-47e8-b5c6-b13139d0cac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t" Nov 03 22:05:14.811818 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:14.811745 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t" Nov 03 22:05:15.108137 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:15.108039 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t"] Nov 03 22:05:15.108285 ip-10-0-131-253 kubenswrapper[5041]: W1103 22:05:15.108143 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11448cf2_8179_47e8_b5c6_b13139d0cac3.slice/crio-676490110b89e8f04247c35613e6b57ee4195646f8d3cc4249e378d5c8c76974 WatchSource:0}: Error finding container 676490110b89e8f04247c35613e6b57ee4195646f8d3cc4249e378d5c8c76974: Status 404 returned error can't find the container with id 676490110b89e8f04247c35613e6b57ee4195646f8d3cc4249e378d5c8c76974 Nov 03 22:05:15.427560 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:15.427528 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t" event={"ID":"11448cf2-8179-47e8-b5c6-b13139d0cac3","Type":"ContainerStarted","Data":"676490110b89e8f04247c35613e6b57ee4195646f8d3cc4249e378d5c8c76974"} Nov 03 22:05:16.436316 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:16.436281 5041 generic.go:356] "Generic (PLEG): container finished" podID="11448cf2-8179-47e8-b5c6-b13139d0cac3" containerID="bc3859f99d8decb768a73fcedeeef5a06065f8ad557df211b0cf80aaf9656b05" exitCode=0 Nov 03 22:05:16.436770 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:16.436358 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t" event={"ID":"11448cf2-8179-47e8-b5c6-b13139d0cac3","Type":"ContainerDied","Data":"bc3859f99d8decb768a73fcedeeef5a06065f8ad557df211b0cf80aaf9656b05"} Nov 03 22:05:17.585228 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:17.585201 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t" Nov 03 22:05:17.705106 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:17.705073 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11448cf2-8179-47e8-b5c6-b13139d0cac3-kube-api-access-qzndr\" (UniqueName: \"kubernetes.io/projected/11448cf2-8179-47e8-b5c6-b13139d0cac3-kube-api-access-qzndr\") pod \"11448cf2-8179-47e8-b5c6-b13139d0cac3\" (UID: \"11448cf2-8179-47e8-b5c6-b13139d0cac3\") " Nov 03 22:05:17.705565 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:17.705124 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11448cf2-8179-47e8-b5c6-b13139d0cac3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11448cf2-8179-47e8-b5c6-b13139d0cac3-maas-api-xks-ca\") pod \"11448cf2-8179-47e8-b5c6-b13139d0cac3\" (UID: \"11448cf2-8179-47e8-b5c6-b13139d0cac3\") " Nov 03 22:05:17.705565 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:17.705161 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11448cf2-8179-47e8-b5c6-b13139d0cac3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11448cf2-8179-47e8-b5c6-b13139d0cac3-maas-api-openshift-ca\") pod \"11448cf2-8179-47e8-b5c6-b13139d0cac3\" (UID: \"11448cf2-8179-47e8-b5c6-b13139d0cac3\") " Nov 03 22:05:17.705684 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:17.705573 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11448cf2-8179-47e8-b5c6-b13139d0cac3-maas-api-openshift-ca" pod "11448cf2-8179-47e8-b5c6-b13139d0cac3" (UID: "11448cf2-8179-47e8-b5c6-b13139d0cac3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:05:17.707239 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:17.707217 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11448cf2-8179-47e8-b5c6-b13139d0cac3-maas-api-xks-ca" pod "11448cf2-8179-47e8-b5c6-b13139d0cac3" (UID: "11448cf2-8179-47e8-b5c6-b13139d0cac3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:05:17.707338 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:17.707237 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11448cf2-8179-47e8-b5c6-b13139d0cac3-kube-api-access-qzndr" pod "11448cf2-8179-47e8-b5c6-b13139d0cac3" (UID: "11448cf2-8179-47e8-b5c6-b13139d0cac3"). InnerVolumeSpecName "kube-api-access-qzndr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:05:17.806857 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:17.806825 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzndr\" (UniqueName: \"kubernetes.io/projected/11448cf2-8179-47e8-b5c6-b13139d0cac3-kube-api-access-qzndr\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:05:17.806857 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:17.806853 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11448cf2-8179-47e8-b5c6-b13139d0cac3-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:05:17.806857 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:17.806864 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11448cf2-8179-47e8-b5c6-b13139d0cac3-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:05:18.449324 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:18.449285 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t" event={"ID":"11448cf2-8179-47e8-b5c6-b13139d0cac3","Type":"ContainerDied","Data":"676490110b89e8f04247c35613e6b57ee4195646f8d3cc4249e378d5c8c76974"} Nov 03 22:05:18.449324 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:18.449314 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t" Nov 03 22:05:18.449555 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:18.449347 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="676490110b89e8f04247c35613e6b57ee4195646f8d3cc4249e378d5c8c76974" Nov 03 22:05:24.884183 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:24.884144 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n"] Nov 03 22:05:24.918960 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:24.918924 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv"] Nov 03 22:05:29.031187 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:29.031150 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn"] Nov 03 22:05:29.044447 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:29.044418 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ecc6d17-8n5wn"] Nov 03 22:05:29.884706 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:29.884642 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" podUID="107e2265-6272-44e7-9ae8-eb86c7038b71" containerName="payload-processing" containerID="cri-o://755961cb4d2079bc261118fefa5dad732fd8fe8f59f749bce08fde1b998a2b99" gracePeriod=25 Nov 03 22:05:29.919369 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:29.919336 5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" podUID="bccca265-6537-43d2-b5c1-d301283c97a2" containerName="payload-pre-processing" containerID="cri-o://a4e641aba60fdb44c19b65e595878c8ad3082612a5b99719bf72e6a5d3b99fc5" gracePeriod=25 Nov 03 22:05:30.199988 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.199957 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" Nov 03 22:05:30.225929 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.225900 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" Nov 03 22:05:30.233586 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.233557 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bccca265-6537-43d2-b5c1-d301283c97a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bccca265-6537-43d2-b5c1-d301283c97a2-plugins-config-volume\") pod \"bccca265-6537-43d2-b5c1-d301283c97a2\" (UID: \"bccca265-6537-43d2-b5c1-d301283c97a2\") " Nov 03 22:05:30.233735 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.233624 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107e2265-6272-44e7-9ae8-eb86c7038b71-kube-api-access-b6lck\" (UniqueName: \"kubernetes.io/projected/107e2265-6272-44e7-9ae8-eb86c7038b71-kube-api-access-b6lck\") pod \"107e2265-6272-44e7-9ae8-eb86c7038b71\" (UID: \"107e2265-6272-44e7-9ae8-eb86c7038b71\") " Nov 03 22:05:30.233782 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.233742 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107e2265-6272-44e7-9ae8-eb86c7038b71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107e2265-6272-44e7-9ae8-eb86c7038b71-plugins-config-volume\") pod \"107e2265-6272-44e7-9ae8-eb86c7038b71\" (UID: \"107e2265-6272-44e7-9ae8-eb86c7038b71\") " Nov 03 22:05:30.233831 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.233791 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bccca265-6537-43d2-b5c1-d301283c97a2-kube-api-access-mkkdw\" (UniqueName: \"kubernetes.io/projected/bccca265-6537-43d2-b5c1-d301283c97a2-kube-api-access-mkkdw\") pod \"bccca265-6537-43d2-b5c1-d301283c97a2\" (UID: \"bccca265-6537-43d2-b5c1-d301283c97a2\") " Nov 03 22:05:30.234067 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.234040 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bccca265-6537-43d2-b5c1-d301283c97a2-plugins-config-volume" pod "bccca265-6537-43d2-b5c1-d301283c97a2" (UID: "bccca265-6537-43d2-b5c1-d301283c97a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:05:30.234157 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.234110 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107e2265-6272-44e7-9ae8-eb86c7038b71-plugins-config-volume" pod "107e2265-6272-44e7-9ae8-eb86c7038b71" (UID: "107e2265-6272-44e7-9ae8-eb86c7038b71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:05:30.234244 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.234227 5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bccca265-6537-43d2-b5c1-d301283c97a2-plugins-config-volume\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:05:30.234289 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.234250 5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107e2265-6272-44e7-9ae8-eb86c7038b71-plugins-config-volume\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:05:30.235747 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.235730 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107e2265-6272-44e7-9ae8-eb86c7038b71-kube-api-access-b6lck" pod "107e2265-6272-44e7-9ae8-eb86c7038b71" (UID: "107e2265-6272-44e7-9ae8-eb86c7038b71"). InnerVolumeSpecName "kube-api-access-b6lck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:05:30.236244 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.236225 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccca265-6537-43d2-b5c1-d301283c97a2-kube-api-access-mkkdw" pod "bccca265-6537-43d2-b5c1-d301283c97a2" (UID: "bccca265-6537-43d2-b5c1-d301283c97a2"). InnerVolumeSpecName "kube-api-access-mkkdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:05:30.334917 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.334883 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6lck\" (UniqueName: \"kubernetes.io/projected/107e2265-6272-44e7-9ae8-eb86c7038b71-kube-api-access-b6lck\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:05:30.334917 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.334914 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkkdw\" (UniqueName: \"kubernetes.io/projected/bccca265-6537-43d2-b5c1-d301283c97a2-kube-api-access-mkkdw\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:05:30.523767 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.523729 5041 generic.go:356] "Generic (PLEG): container finished" podID="bccca265-6537-43d2-b5c1-d301283c97a2" containerID="a4e641aba60fdb44c19b65e595878c8ad3082612a5b99719bf72e6a5d3b99fc5" exitCode=0 Nov 03 22:05:30.523981 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.523792 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" Nov 03 22:05:30.523981 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.523803 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" event={"ID":"bccca265-6537-43d2-b5c1-d301283c97a2","Type":"ContainerDied","Data":"a4e641aba60fdb44c19b65e595878c8ad3082612a5b99719bf72e6a5d3b99fc5"} Nov 03 22:05:30.523981 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.523893 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv" event={"ID":"bccca265-6537-43d2-b5c1-d301283c97a2","Type":"ContainerDied","Data":"e9361fa0113159a257754803d3f84f9834a5943392bee7dc37545f94ebe3a439"} Nov 03 22:05:30.523981 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.523921 5041 scope.go:122] "RemoveContainer" containerID="a4e641aba60fdb44c19b65e595878c8ad3082612a5b99719bf72e6a5d3b99fc5" Nov 03 22:05:30.525366 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.525340 5041 generic.go:356] "Generic (PLEG): container finished" podID="107e2265-6272-44e7-9ae8-eb86c7038b71" containerID="755961cb4d2079bc261118fefa5dad732fd8fe8f59f749bce08fde1b998a2b99" exitCode=0 Nov 03 22:05:30.525521 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.525418 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" Nov 03 22:05:30.525521 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.525432 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" event={"ID":"107e2265-6272-44e7-9ae8-eb86c7038b71","Type":"ContainerDied","Data":"755961cb4d2079bc261118fefa5dad732fd8fe8f59f749bce08fde1b998a2b99"} Nov 03 22:05:30.525521 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.525485 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n" event={"ID":"107e2265-6272-44e7-9ae8-eb86c7038b71","Type":"ContainerDied","Data":"96fa0d799c68c686b3e9d2b1c96d591b1bef27ab5f4c25c840df2d88a08c4ffd"} Nov 03 22:05:30.534118 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.533878 5041 scope.go:122] "RemoveContainer" containerID="a4e641aba60fdb44c19b65e595878c8ad3082612a5b99719bf72e6a5d3b99fc5" Nov 03 22:05:30.534244 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:05:30.534199 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e641aba60fdb44c19b65e595878c8ad3082612a5b99719bf72e6a5d3b99fc5\": container with ID starting with a4e641aba60fdb44c19b65e595878c8ad3082612a5b99719bf72e6a5d3b99fc5 not found: ID does not exist" containerID="a4e641aba60fdb44c19b65e595878c8ad3082612a5b99719bf72e6a5d3b99fc5" Nov 03 22:05:30.534298 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.534245 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e641aba60fdb44c19b65e595878c8ad3082612a5b99719bf72e6a5d3b99fc5"} err="failed to get container status \"a4e641aba60fdb44c19b65e595878c8ad3082612a5b99719bf72e6a5d3b99fc5\": rpc error: code = NotFound desc = could not find container \"a4e641aba60fdb44c19b65e595878c8ad3082612a5b99719bf72e6a5d3b99fc5\": container with ID starting with a4e641aba60fdb44c19b65e595878c8ad3082612a5b99719bf72e6a5d3b99fc5 not found: ID does not exist" Nov 03 22:05:30.534298 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.534271 5041 scope.go:122] "RemoveContainer" containerID="755961cb4d2079bc261118fefa5dad732fd8fe8f59f749bce08fde1b998a2b99" Nov 03 22:05:30.543610 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.543593 5041 scope.go:122] "RemoveContainer" containerID="755961cb4d2079bc261118fefa5dad732fd8fe8f59f749bce08fde1b998a2b99" Nov 03 22:05:30.543893 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:05:30.543873 5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"755961cb4d2079bc261118fefa5dad732fd8fe8f59f749bce08fde1b998a2b99\": container with ID starting with 755961cb4d2079bc261118fefa5dad732fd8fe8f59f749bce08fde1b998a2b99 not found: ID does not exist" containerID="755961cb4d2079bc261118fefa5dad732fd8fe8f59f749bce08fde1b998a2b99" Nov 03 22:05:30.543951 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.543900 5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755961cb4d2079bc261118fefa5dad732fd8fe8f59f749bce08fde1b998a2b99"} err="failed to get container status \"755961cb4d2079bc261118fefa5dad732fd8fe8f59f749bce08fde1b998a2b99\": rpc error: code = NotFound desc = could not find container \"755961cb4d2079bc261118fefa5dad732fd8fe8f59f749bce08fde1b998a2b99\": container with ID starting with 755961cb4d2079bc261118fefa5dad732fd8fe8f59f749bce08fde1b998a2b99 not found: ID does not exist" Nov 03 22:05:30.579199 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.579170 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv"] Nov 03 22:05:30.592790 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.592758 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-6a697e-db46d9c5-7rdsv"] Nov 03 22:05:30.621593 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.621565 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1082de9c-a6d2-4e01-bc68-385d5d019ce7" path="/var/lib/kubelet/pods/1082de9c-a6d2-4e01-bc68-385d5d019ce7/volumes" Nov 03 22:05:30.622021 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.622006 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bccca265-6537-43d2-b5c1-d301283c97a2" path="/var/lib/kubelet/pods/bccca265-6537-43d2-b5c1-d301283c97a2/volumes" Nov 03 22:05:30.622312 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.622299 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n"] Nov 03 22:05:30.623932 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:30.623911 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-6a697e-598df485bd-kv46n"] Nov 03 22:05:32.349496 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:32.349456 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t"] Nov 03 22:05:32.362718 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:32.362688 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-6a697e-g6m2t"] Nov 03 22:05:32.621712 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:32.621620 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107e2265-6272-44e7-9ae8-eb86c7038b71" path="/var/lib/kubelet/pods/107e2265-6272-44e7-9ae8-eb86c7038b71/volumes" Nov 03 22:05:32.622065 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:32.622051 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11448cf2-8179-47e8-b5c6-b13139d0cac3" path="/var/lib/kubelet/pods/11448cf2-8179-47e8-b5c6-b13139d0cac3/volumes" Nov 03 22:05:40.777427 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:05:40.777383 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:05:49.583962 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:49.583922 5041 scope.go:122] "RemoveContainer" containerID="ce19e3bd5855eebe8bf5a26672e5f4b43c4d782ac6d3e601af0028c364310b1e" Nov 03 22:05:49.593330 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:49.593306 5041 scope.go:122] "RemoveContainer" containerID="6a2a20c9ab1caa13a94624a9e2945bdaf3d19e55a91922d6d6763356a9645142" Nov 03 22:05:56.399930 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.399893 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2"] Nov 03 22:05:56.400400 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.400385 5041 cpu_manager.go:414] "containerMap: removing container" podUID="107e2265-6272-44e7-9ae8-eb86c7038b71" containerName="payload-processing" Nov 03 22:05:56.400446 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.400401 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107e2265-6272-44e7-9ae8-eb86c7038b71" containerName="payload-processing" Nov 03 22:05:56.400446 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.400421 5041 cpu_manager.go:414] "containerMap: removing container" podUID="bccca265-6537-43d2-b5c1-d301283c97a2" containerName="payload-pre-processing" Nov 03 22:05:56.400446 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.400426 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bccca265-6537-43d2-b5c1-d301283c97a2" containerName="payload-pre-processing" Nov 03 22:05:56.400555 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.400445 5041 cpu_manager.go:414] "containerMap: removing container" podUID="11448cf2-8179-47e8-b5c6-b13139d0cac3" containerName="revoke-keys" Nov 03 22:05:56.400555 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.400453 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11448cf2-8179-47e8-b5c6-b13139d0cac3" containerName="revoke-keys" Nov 03 22:05:56.400555 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.400551 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bccca265-6537-43d2-b5c1-d301283c97a2" containerName="payload-pre-processing" Nov 03 22:05:56.400670 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.400562 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="11448cf2-8179-47e8-b5c6-b13139d0cac3" containerName="revoke-keys" Nov 03 22:05:56.400670 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.400573 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="107e2265-6272-44e7-9ae8-eb86c7038b71" containerName="payload-processing" Nov 03 22:05:56.405307 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.405291 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" Nov 03 22:05:56.419679 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.419631 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43e22ae4-2e09-4039-ab87-32057416b1ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-4qvl2\" (UID: \"43e22ae4-2e09-4039-ab87-32057416b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" Nov 03 22:05:56.419829 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.419710 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43e22ae4-2e09-4039-ab87-32057416b1ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-4qvl2\" (UID: \"43e22ae4-2e09-4039-ab87-32057416b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" Nov 03 22:05:56.419829 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.419793 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsl2m\" (UniqueName: \"kubernetes.io/projected/43e22ae4-2e09-4039-ab87-32057416b1ad-kube-api-access-jsl2m\") pod \"maas-api-revoke-sub-f03629db-4qvl2\" (UID: \"43e22ae4-2e09-4039-ab87-32057416b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" Nov 03 22:05:56.424986 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.424956 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2"] Nov 03 22:05:56.432637 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.432611 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:05:56.432818 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.432796 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n4fdx\"" Nov 03 22:05:56.520742 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.520708 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43e22ae4-2e09-4039-ab87-32057416b1ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-4qvl2\" (UID: \"43e22ae4-2e09-4039-ab87-32057416b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" Nov 03 22:05:56.520927 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.520751 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43e22ae4-2e09-4039-ab87-32057416b1ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-4qvl2\" (UID: \"43e22ae4-2e09-4039-ab87-32057416b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" Nov 03 22:05:56.520927 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.520791 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsl2m\" (UniqueName: \"kubernetes.io/projected/43e22ae4-2e09-4039-ab87-32057416b1ad-kube-api-access-jsl2m\") pod \"maas-api-revoke-sub-f03629db-4qvl2\" (UID: \"43e22ae4-2e09-4039-ab87-32057416b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" Nov 03 22:05:56.521432 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.521404 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43e22ae4-2e09-4039-ab87-32057416b1ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-4qvl2\" (UID: \"43e22ae4-2e09-4039-ab87-32057416b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" Nov 03 22:05:56.523174 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.523145 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43e22ae4-2e09-4039-ab87-32057416b1ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-4qvl2\" (UID: \"43e22ae4-2e09-4039-ab87-32057416b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" Nov 03 22:05:56.541433 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.541412 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsl2m\" (UniqueName: \"kubernetes.io/projected/43e22ae4-2e09-4039-ab87-32057416b1ad-kube-api-access-jsl2m\") pod \"maas-api-revoke-sub-f03629db-4qvl2\" (UID: \"43e22ae4-2e09-4039-ab87-32057416b1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" Nov 03 22:05:56.726100 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:56.726072 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" Nov 03 22:05:57.018978 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:57.018952 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2"] Nov 03 22:05:57.019155 ip-10-0-131-253 kubenswrapper[5041]: W1103 22:05:57.019132 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e22ae4_2e09_4039_ab87_32057416b1ad.slice/crio-598e0e02f7e38713d2013674f1225d226a6d65dd0dd1cde352e94bf30e43fe8b WatchSource:0}: Error finding container 598e0e02f7e38713d2013674f1225d226a6d65dd0dd1cde352e94bf30e43fe8b: Status 404 returned error can't find the container with id 598e0e02f7e38713d2013674f1225d226a6d65dd0dd1cde352e94bf30e43fe8b Nov 03 22:05:57.817124 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:57.817081 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" event={"ID":"43e22ae4-2e09-4039-ab87-32057416b1ad","Type":"ContainerStarted","Data":"88d8c0e27e7e962b38bb5113c279e97d8847b2d5ff85673a2df2f2f09e703f17"} Nov 03 22:05:57.817594 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:57.817153 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" event={"ID":"43e22ae4-2e09-4039-ab87-32057416b1ad","Type":"ContainerStarted","Data":"598e0e02f7e38713d2013674f1225d226a6d65dd0dd1cde352e94bf30e43fe8b"} Nov 03 22:05:57.858596 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:57.858540 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" podStartSLOduration=1.858522521 podStartE2EDuration="1.858522521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:57.857202816 +0000 UTC m=+2173.364431115" watchObservedRunningTime="2026-11-03 22:05:57.858522521 +0000 UTC m=+2173.365750802" Nov 03 22:05:58.824643 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:58.824605 5041 generic.go:356] "Generic (PLEG): container finished" podID="43e22ae4-2e09-4039-ab87-32057416b1ad" containerID="88d8c0e27e7e962b38bb5113c279e97d8847b2d5ff85673a2df2f2f09e703f17" exitCode=0 Nov 03 22:05:58.825134 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:58.824687 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" event={"ID":"43e22ae4-2e09-4039-ab87-32057416b1ad","Type":"ContainerDied","Data":"88d8c0e27e7e962b38bb5113c279e97d8847b2d5ff85673a2df2f2f09e703f17"} Nov 03 22:05:59.983140 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:05:59.983111 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" Nov 03 22:06:00.061876 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:00.061835 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e22ae4-2e09-4039-ab87-32057416b1ad-kube-api-access-jsl2m\" (UniqueName: \"kubernetes.io/projected/43e22ae4-2e09-4039-ab87-32057416b1ad-kube-api-access-jsl2m\") pod \"43e22ae4-2e09-4039-ab87-32057416b1ad\" (UID: \"43e22ae4-2e09-4039-ab87-32057416b1ad\") " Nov 03 22:06:00.062073 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:00.061989 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43e22ae4-2e09-4039-ab87-32057416b1ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43e22ae4-2e09-4039-ab87-32057416b1ad-maas-api-openshift-ca\") pod \"43e22ae4-2e09-4039-ab87-32057416b1ad\" (UID: \"43e22ae4-2e09-4039-ab87-32057416b1ad\") " Nov 03 22:06:00.062073 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:00.062045 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43e22ae4-2e09-4039-ab87-32057416b1ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43e22ae4-2e09-4039-ab87-32057416b1ad-maas-api-xks-ca\") pod \"43e22ae4-2e09-4039-ab87-32057416b1ad\" (UID: \"43e22ae4-2e09-4039-ab87-32057416b1ad\") " Nov 03 22:06:00.062418 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:00.062383 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e22ae4-2e09-4039-ab87-32057416b1ad-maas-api-openshift-ca" pod "43e22ae4-2e09-4039-ab87-32057416b1ad" (UID: "43e22ae4-2e09-4039-ab87-32057416b1ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:06:00.062630 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:00.062605 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43e22ae4-2e09-4039-ab87-32057416b1ad-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:06:00.064208 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:00.064184 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e22ae4-2e09-4039-ab87-32057416b1ad-maas-api-xks-ca" pod "43e22ae4-2e09-4039-ab87-32057416b1ad" (UID: "43e22ae4-2e09-4039-ab87-32057416b1ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:06:00.064296 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:00.064239 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e22ae4-2e09-4039-ab87-32057416b1ad-kube-api-access-jsl2m" pod "43e22ae4-2e09-4039-ab87-32057416b1ad" (UID: "43e22ae4-2e09-4039-ab87-32057416b1ad"). InnerVolumeSpecName "kube-api-access-jsl2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:06:00.163498 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:00.163420 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43e22ae4-2e09-4039-ab87-32057416b1ad-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:06:00.163498 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:00.163449 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsl2m\" (UniqueName: \"kubernetes.io/projected/43e22ae4-2e09-4039-ab87-32057416b1ad-kube-api-access-jsl2m\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:06:00.838499 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:00.838471 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" Nov 03 22:06:00.838698 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:00.838466 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2" event={"ID":"43e22ae4-2e09-4039-ab87-32057416b1ad","Type":"ContainerDied","Data":"598e0e02f7e38713d2013674f1225d226a6d65dd0dd1cde352e94bf30e43fe8b"} Nov 03 22:06:00.838698 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:00.838629 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="598e0e02f7e38713d2013674f1225d226a6d65dd0dd1cde352e94bf30e43fe8b" Nov 03 22:06:10.791255 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:06:10.791207 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:06:13.921380 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:13.921346 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw"] Nov 03 22:06:13.921968 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:13.921952 5041 cpu_manager.go:414] "containerMap: removing container" podUID="43e22ae4-2e09-4039-ab87-32057416b1ad" containerName="revoke-keys" Nov 03 22:06:13.922019 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:13.921970 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e22ae4-2e09-4039-ab87-32057416b1ad" containerName="revoke-keys" Nov 03 22:06:13.922057 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:13.922048 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e22ae4-2e09-4039-ab87-32057416b1ad" containerName="revoke-keys" Nov 03 22:06:13.925383 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:13.925367 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" Nov 03 22:06:13.943294 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:13.943268 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw"] Nov 03 22:06:13.953209 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:13.953176 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:06:13.953375 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:13.953335 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n4fdx\"" Nov 03 22:06:14.048956 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:14.048916 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbd78\" (UniqueName: \"kubernetes.io/projected/f98da404-a032-48b7-8561-92d605ead3c7-kube-api-access-wbd78\") pod \"maas-api-revoke-sub-0165d7d2-rplcw\" (UID: \"f98da404-a032-48b7-8561-92d605ead3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" Nov 03 22:06:14.049126 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:14.049096 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98da404-a032-48b7-8561-92d605ead3c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rplcw\" (UID: \"f98da404-a032-48b7-8561-92d605ead3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" Nov 03 22:06:14.049171 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:14.049142 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98da404-a032-48b7-8561-92d605ead3c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rplcw\" (UID: \"f98da404-a032-48b7-8561-92d605ead3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" Nov 03 22:06:14.150463 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:14.150425 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98da404-a032-48b7-8561-92d605ead3c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rplcw\" (UID: \"f98da404-a032-48b7-8561-92d605ead3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" Nov 03 22:06:14.150709 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:14.150473 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98da404-a032-48b7-8561-92d605ead3c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rplcw\" (UID: \"f98da404-a032-48b7-8561-92d605ead3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" Nov 03 22:06:14.150709 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:14.150525 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbd78\" (UniqueName: \"kubernetes.io/projected/f98da404-a032-48b7-8561-92d605ead3c7-kube-api-access-wbd78\") pod \"maas-api-revoke-sub-0165d7d2-rplcw\" (UID: \"f98da404-a032-48b7-8561-92d605ead3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" Nov 03 22:06:14.151199 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:14.151168 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98da404-a032-48b7-8561-92d605ead3c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rplcw\" (UID: \"f98da404-a032-48b7-8561-92d605ead3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" Nov 03 22:06:14.152891 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:14.152872 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98da404-a032-48b7-8561-92d605ead3c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rplcw\" (UID: \"f98da404-a032-48b7-8561-92d605ead3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" Nov 03 22:06:14.170750 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:14.170723 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbd78\" (UniqueName: \"kubernetes.io/projected/f98da404-a032-48b7-8561-92d605ead3c7-kube-api-access-wbd78\") pod \"maas-api-revoke-sub-0165d7d2-rplcw\" (UID: \"f98da404-a032-48b7-8561-92d605ead3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" Nov 03 22:06:14.239144 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:14.239114 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" Nov 03 22:06:14.533506 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:14.533480 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw"] Nov 03 22:06:14.533639 ip-10-0-131-253 kubenswrapper[5041]: W1103 22:06:14.533541 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98da404_a032_48b7_8561_92d605ead3c7.slice/crio-f4387212d390463bf46f63ce4407582f0329f6e9d9fa5c8b983b4b7507f154d6 WatchSource:0}: Error finding container f4387212d390463bf46f63ce4407582f0329f6e9d9fa5c8b983b4b7507f154d6: Status 404 returned error can't find the container with id f4387212d390463bf46f63ce4407582f0329f6e9d9fa5c8b983b4b7507f154d6 Nov 03 22:06:14.948597 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:14.948503 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" event={"ID":"f98da404-a032-48b7-8561-92d605ead3c7","Type":"ContainerStarted","Data":"bcb37777b61a6e8b9b757973645913ca7168c6c6a437d84c755a6748aeaf0fb9"} Nov 03 22:06:14.948597 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:14.948557 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" event={"ID":"f98da404-a032-48b7-8561-92d605ead3c7","Type":"ContainerStarted","Data":"f4387212d390463bf46f63ce4407582f0329f6e9d9fa5c8b983b4b7507f154d6"} Nov 03 22:06:14.991364 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:14.991311 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" podStartSLOduration=1.991297653 podStartE2EDuration="1.991297653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:14.988079713 +0000 UTC m=+2190.495308011" watchObservedRunningTime="2026-11-03 22:06:14.991297653 +0000 UTC m=+2190.498525963" Nov 03 22:06:15.955995 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:15.955953 5041 generic.go:356] "Generic (PLEG): container finished" podID="f98da404-a032-48b7-8561-92d605ead3c7" containerID="bcb37777b61a6e8b9b757973645913ca7168c6c6a437d84c755a6748aeaf0fb9" exitCode=0 Nov 03 22:06:15.956434 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:15.956042 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" event={"ID":"f98da404-a032-48b7-8561-92d605ead3c7","Type":"ContainerDied","Data":"bcb37777b61a6e8b9b757973645913ca7168c6c6a437d84c755a6748aeaf0fb9"} Nov 03 22:06:17.109630 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:17.109600 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" Nov 03 22:06:17.180735 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:17.180701 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98da404-a032-48b7-8561-92d605ead3c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98da404-a032-48b7-8561-92d605ead3c7-maas-api-openshift-ca\") pod \"f98da404-a032-48b7-8561-92d605ead3c7\" (UID: \"f98da404-a032-48b7-8561-92d605ead3c7\") " Nov 03 22:06:17.180903 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:17.180822 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98da404-a032-48b7-8561-92d605ead3c7-kube-api-access-wbd78\" (UniqueName: \"kubernetes.io/projected/f98da404-a032-48b7-8561-92d605ead3c7-kube-api-access-wbd78\") pod \"f98da404-a032-48b7-8561-92d605ead3c7\" (UID: \"f98da404-a032-48b7-8561-92d605ead3c7\") " Nov 03 22:06:17.180903 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:17.180882 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f98da404-a032-48b7-8561-92d605ead3c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98da404-a032-48b7-8561-92d605ead3c7-maas-api-xks-ca\") pod \"f98da404-a032-48b7-8561-92d605ead3c7\" (UID: \"f98da404-a032-48b7-8561-92d605ead3c7\") " Nov 03 22:06:17.181141 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:17.181116 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98da404-a032-48b7-8561-92d605ead3c7-maas-api-openshift-ca" pod "f98da404-a032-48b7-8561-92d605ead3c7" (UID: "f98da404-a032-48b7-8561-92d605ead3c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:06:17.181314 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:17.181299 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98da404-a032-48b7-8561-92d605ead3c7-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:06:17.183422 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:17.183396 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98da404-a032-48b7-8561-92d605ead3c7-maas-api-xks-ca" pod "f98da404-a032-48b7-8561-92d605ead3c7" (UID: "f98da404-a032-48b7-8561-92d605ead3c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:06:17.183422 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:17.183415 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98da404-a032-48b7-8561-92d605ead3c7-kube-api-access-wbd78" pod "f98da404-a032-48b7-8561-92d605ead3c7" (UID: "f98da404-a032-48b7-8561-92d605ead3c7"). InnerVolumeSpecName "kube-api-access-wbd78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:06:17.282697 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:17.282608 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbd78\" (UniqueName: \"kubernetes.io/projected/f98da404-a032-48b7-8561-92d605ead3c7-kube-api-access-wbd78\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:06:17.282697 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:17.282637 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98da404-a032-48b7-8561-92d605ead3c7-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:06:17.977810 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:17.977780 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" Nov 03 22:06:17.977985 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:17.977778 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw" event={"ID":"f98da404-a032-48b7-8561-92d605ead3c7","Type":"ContainerDied","Data":"f4387212d390463bf46f63ce4407582f0329f6e9d9fa5c8b983b4b7507f154d6"} Nov 03 22:06:17.977985 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:17.977931 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4387212d390463bf46f63ce4407582f0329f6e9d9fa5c8b983b4b7507f154d6" Nov 03 22:06:24.931618 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:24.931569 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg"] Nov 03 22:06:24.932162 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:24.932082 5041 cpu_manager.go:414] "containerMap: removing container" podUID="f98da404-a032-48b7-8561-92d605ead3c7" containerName="revoke-keys" Nov 03 22:06:24.932162 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:24.932095 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98da404-a032-48b7-8561-92d605ead3c7" containerName="revoke-keys" Nov 03 22:06:24.932292 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:24.932183 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98da404-a032-48b7-8561-92d605ead3c7" containerName="revoke-keys" Nov 03 22:06:24.935858 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:24.935827 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" Nov 03 22:06:24.953591 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:24.953561 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg"] Nov 03 22:06:24.962423 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:24.962398 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:06:24.962569 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:24.962465 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n4fdx\"" Nov 03 22:06:25.056423 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:25.056391 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db307e5d-5d16-4006-a583-56b0988788f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xkkmg\" (UID: \"db307e5d-5d16-4006-a583-56b0988788f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" Nov 03 22:06:25.056684 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:25.056568 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db307e5d-5d16-4006-a583-56b0988788f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xkkmg\" (UID: \"db307e5d-5d16-4006-a583-56b0988788f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" Nov 03 22:06:25.056684 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:25.056631 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8fb4\" (UniqueName: \"kubernetes.io/projected/db307e5d-5d16-4006-a583-56b0988788f4-kube-api-access-r8fb4\") pod \"maas-api-revoke-sub-6b19338f-xkkmg\" (UID: \"db307e5d-5d16-4006-a583-56b0988788f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" Nov 03 22:06:25.158068 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:25.158028 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8fb4\" (UniqueName: \"kubernetes.io/projected/db307e5d-5d16-4006-a583-56b0988788f4-kube-api-access-r8fb4\") pod \"maas-api-revoke-sub-6b19338f-xkkmg\" (UID: \"db307e5d-5d16-4006-a583-56b0988788f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" Nov 03 22:06:25.158293 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:25.158121 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db307e5d-5d16-4006-a583-56b0988788f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xkkmg\" (UID: \"db307e5d-5d16-4006-a583-56b0988788f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" Nov 03 22:06:25.158293 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:25.158253 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db307e5d-5d16-4006-a583-56b0988788f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xkkmg\" (UID: \"db307e5d-5d16-4006-a583-56b0988788f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" Nov 03 22:06:25.159004 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:25.158975 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db307e5d-5d16-4006-a583-56b0988788f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xkkmg\" (UID: \"db307e5d-5d16-4006-a583-56b0988788f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" Nov 03 22:06:25.161068 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:25.161036 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db307e5d-5d16-4006-a583-56b0988788f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xkkmg\" (UID: \"db307e5d-5d16-4006-a583-56b0988788f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" Nov 03 22:06:25.178963 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:25.178936 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8fb4\" (UniqueName: \"kubernetes.io/projected/db307e5d-5d16-4006-a583-56b0988788f4-kube-api-access-r8fb4\") pod \"maas-api-revoke-sub-6b19338f-xkkmg\" (UID: \"db307e5d-5d16-4006-a583-56b0988788f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" Nov 03 22:06:25.257829 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:25.257790 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" Nov 03 22:06:25.557896 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:25.557865 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg"] Nov 03 22:06:26.026688 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:26.026633 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" event={"ID":"db307e5d-5d16-4006-a583-56b0988788f4","Type":"ContainerStarted","Data":"7cbca67ae702c54e9b341e85cfc04078c628fb521a2b17e012dd984d33be6379"} Nov 03 22:06:26.027180 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:26.026712 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" event={"ID":"db307e5d-5d16-4006-a583-56b0988788f4","Type":"ContainerStarted","Data":"bc2534ad96ee0b3b17994057c16a2d8c45aea274729c35dfa8bf543d723df4be"} Nov 03 22:06:26.069185 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:26.069131 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" podStartSLOduration=2.069117328 podStartE2EDuration="2.069117328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:26.066537072 +0000 UTC m=+2201.573765387" watchObservedRunningTime="2026-11-03 22:06:26.069117328 +0000 UTC m=+2201.576345602" Nov 03 22:06:27.034064 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:27.034033 5041 generic.go:356] "Generic (PLEG): container finished" podID="db307e5d-5d16-4006-a583-56b0988788f4" containerID="7cbca67ae702c54e9b341e85cfc04078c628fb521a2b17e012dd984d33be6379" exitCode=0 Nov 03 22:06:27.034502 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:27.034116 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" event={"ID":"db307e5d-5d16-4006-a583-56b0988788f4","Type":"ContainerDied","Data":"7cbca67ae702c54e9b341e85cfc04078c628fb521a2b17e012dd984d33be6379"} Nov 03 22:06:28.183994 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:28.183966 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" Nov 03 22:06:28.288768 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:28.288726 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db307e5d-5d16-4006-a583-56b0988788f4-kube-api-access-r8fb4\" (UniqueName: \"kubernetes.io/projected/db307e5d-5d16-4006-a583-56b0988788f4-kube-api-access-r8fb4\") pod \"db307e5d-5d16-4006-a583-56b0988788f4\" (UID: \"db307e5d-5d16-4006-a583-56b0988788f4\") " Nov 03 22:06:28.288927 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:28.288870 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db307e5d-5d16-4006-a583-56b0988788f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db307e5d-5d16-4006-a583-56b0988788f4-maas-api-openshift-ca\") pod \"db307e5d-5d16-4006-a583-56b0988788f4\" (UID: \"db307e5d-5d16-4006-a583-56b0988788f4\") " Nov 03 22:06:28.288927 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:28.288918 5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db307e5d-5d16-4006-a583-56b0988788f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db307e5d-5d16-4006-a583-56b0988788f4-maas-api-xks-ca\") pod \"db307e5d-5d16-4006-a583-56b0988788f4\" (UID: \"db307e5d-5d16-4006-a583-56b0988788f4\") " Nov 03 22:06:28.289234 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:28.289207 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db307e5d-5d16-4006-a583-56b0988788f4-maas-api-openshift-ca" pod "db307e5d-5d16-4006-a583-56b0988788f4" (UID: "db307e5d-5d16-4006-a583-56b0988788f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:06:28.290870 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:28.290846 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db307e5d-5d16-4006-a583-56b0988788f4-kube-api-access-r8fb4" pod "db307e5d-5d16-4006-a583-56b0988788f4" (UID: "db307e5d-5d16-4006-a583-56b0988788f4"). InnerVolumeSpecName "kube-api-access-r8fb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:06:28.290993 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:28.290894 5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db307e5d-5d16-4006-a583-56b0988788f4-maas-api-xks-ca" pod "db307e5d-5d16-4006-a583-56b0988788f4" (UID: "db307e5d-5d16-4006-a583-56b0988788f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:06:28.390581 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:28.390553 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db307e5d-5d16-4006-a583-56b0988788f4-maas-api-openshift-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:06:28.390581 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:28.390578 5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db307e5d-5d16-4006-a583-56b0988788f4-maas-api-xks-ca\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:06:28.390772 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:28.390592 5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8fb4\" (UniqueName: \"kubernetes.io/projected/db307e5d-5d16-4006-a583-56b0988788f4-kube-api-access-r8fb4\") on node \"ip-10-0-131-253.ec2.internal\" DevicePath \"\"" Nov 03 22:06:29.048729 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:29.048693 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" event={"ID":"db307e5d-5d16-4006-a583-56b0988788f4","Type":"ContainerDied","Data":"bc2534ad96ee0b3b17994057c16a2d8c45aea274729c35dfa8bf543d723df4be"} Nov 03 22:06:29.048729 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:29.048712 5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xkkmg" Nov 03 22:06:29.048989 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:29.048741 5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc2534ad96ee0b3b17994057c16a2d8c45aea274729c35dfa8bf543d723df4be" Nov 03 22:06:40.776467 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:06:40.776430 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:06:41.028174 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:41.028092 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l"] Nov 03 22:06:41.041584 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:41.041548 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nmg5l"] Nov 03 22:06:42.620558 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:42.620510 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de52a474-5c45-4c1e-9a1e-c204a154d00c" path="/var/lib/kubelet/pods/de52a474-5c45-4c1e-9a1e-c204a154d00c/volumes" Nov 03 22:06:47.094109 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:47.094076 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lrr9z_416c8af4-2b21-4015-9cda-e4a7f75d8506/manager/0.log" Nov 03 22:06:47.295635 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:47.295603 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kvrbr_e83c023b-6164-4f13-89bd-9553952667b7/manager/0.log" Nov 03 22:06:47.821634 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:47.821599 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d575fdb56-jwwm2_89259ba9-9d01-4e60-a875-9b02fb8bd98f/manager/0.log" Nov 03 22:06:48.057435 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:48.057408 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vppdx_a8f799a1-8427-44a3-a495-74720e82292a/server/0.log" Nov 03 22:06:48.235598 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:48.235565 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2cctc_8116c434-849d-4d12-8d67-9278246bb78b/manager/0.log" Nov 03 22:06:48.448513 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:48.448484 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-95xs7_8c1dc05e-ba05-4715-9236-1eabf21fb915/proxy/0.log" Nov 03 22:06:49.562408 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:49.562374 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-rplcw_f98da404-a032-48b7-8561-92d605ead3c7/revoke-keys/0.log" Nov 03 22:06:49.635844 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:49.635813 5041 scope.go:122] "RemoveContainer" containerID="a37055562d355e11fee252850eb35e7c608afe431e0ddbabf8dba4248f360e5e" Nov 03 22:06:49.648205 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:49.648174 5041 scope.go:122] "RemoveContainer" containerID="7a3b4e067a4334d232edf667aab0e6ce9f94925f56ca17d2e65f62c679263fe5" Nov 03 22:06:49.743013 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:49.742978 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-cgmkk_6715bed2-8392-4ccb-aa31-ee9690099b30/revoke-keys/0.log" Nov 03 22:06:49.936258 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:49.936175 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-56327c97-z4lt4_2454f93f-f6f2-43bc-8e55-c4974ac964ec/revoke-keys/0.log" Nov 03 22:06:50.121859 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:50.121830 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-xkkmg_db307e5d-5d16-4006-a583-56b0988788f4/revoke-keys/0.log" Nov 03 22:06:50.313576 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:50.313544 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-4qvl2_43e22ae4-2e09-4039-ab87-32057416b1ad/revoke-keys/0.log" Nov 03 22:06:50.506769 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:06:50.506740 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-hzchq_c7500df4-c2fc-47e5-bcbe-330cf017db34/postgres/0.log" Nov 03 22:07:03.026409 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:03.026373 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk"] Nov 03 22:07:03.028928 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:03.028896 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-cgmkk"] Nov 03 22:07:04.622862 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:04.621280 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6715bed2-8392-4ccb-aa31-ee9690099b30" path="/var/lib/kubelet/pods/6715bed2-8392-4ccb-aa31-ee9690099b30/volumes" Nov 03 22:07:10.778100 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:07:10.778061 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:07:12.281249 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:12.281216 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-c446k_b3c86a2a-007c-4a88-87cc-5a97de621e31/manager/0.log" Nov 03 22:07:12.493866 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:12.493830 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6t5ln_f03cef83-f82a-4b4c-901c-e091f6fc58dd/manager/0.log" Nov 03 22:07:12.890589 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:12.890557 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6s2gm_c5b685e7-1863-4500-a5f6-1e58b8335e76/manager/0.log" Nov 03 22:07:13.334027 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:13.333997 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ghgpz_418620a8-7e15-42ac-97fa-ad9568954b5f/manager/0.log" Nov 03 22:07:14.096929 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:14.096900 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs_982671c7-73d5-48b1-9067-d644e1b956d4/istio-proxy/0.log" Nov 03 22:07:14.906545 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:14.906511 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-6kgrf_bd662fa8-196b-4cad-b414-1bc2dbd68aa1/istio-proxy/1.log" Nov 03 22:07:15.117066 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:15.117037 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-8b9jr_69cc2bab-a8e9-4a4a-a114-88663140640d/istio-proxy/0.log" Nov 03 22:07:25.934924 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:25.934886 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-lt4j4_9be23a49-ab91-409d-b131-cb4a917a58e3/istio-proxy/0.log" Nov 03 22:07:26.137985 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:26.137943 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-wc8rh_44bc42a8-430e-44ec-b717-d1a2436968f5/istio-proxy/0.log" Nov 03 22:07:26.929187 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:26.929155 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8_1d77b6f1-de97-4e82-9fb6-0735930827a2/istio-proxy/0.log" Nov 03 22:07:27.109192 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:27.109155 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zlxvh_e63ff686-c19d-485e-b503-22cc297aa852/payload-pre-processing/0.log" Nov 03 22:07:27.502272 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:27.502243 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9f5c5b5d5-hpzcq_0852ff0f-d078-452f-bb56-3c13512f57ce/router/0.log" Nov 03 22:07:27.945989 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:27.945911 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-6kgrf_bd662fa8-196b-4cad-b414-1bc2dbd68aa1/istio-proxy/0.log" Nov 03 22:07:28.694288 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:28.694258 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-bqstk_2f3d92ef-2874-44da-90a5-ff115f8c2755/main/0.log" Nov 03 22:07:28.875204 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:28.875179 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-v2pwb_fa892a3f-4760-45ff-b00b-0840970540bc/main/0.log" Nov 03 22:07:29.289464 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:29.289427 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r_1d7a34c6-9da5-45de-9373-50b3b18eb7b1/main/0.log" Nov 03 22:07:29.651098 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:29.651024 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db_6fe7207f-2102-4312-b28d-9c91c22b0ddb/main/0.log" Nov 03 22:07:33.027614 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:33.027579 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4"] Nov 03 22:07:33.041410 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:33.041377 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56327c97-z4lt4"] Nov 03 22:07:34.621054 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:34.621020 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2454f93f-f6f2-43bc-8e55-c4974ac964ec" path="/var/lib/kubelet/pods/2454f93f-f6f2-43bc-8e55-c4974ac964ec/volumes" Nov 03 22:07:40.025941 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:40.025910 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lrr9z_416c8af4-2b21-4015-9cda-e4a7f75d8506/manager/0.log" Nov 03 22:07:40.213222 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:40.213182 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-kvrbr_e83c023b-6164-4f13-89bd-9553952667b7/manager/0.log" Nov 03 22:07:40.683329 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:40.683302 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d575fdb56-jwwm2_89259ba9-9d01-4e60-a875-9b02fb8bd98f/manager/0.log" Nov 03 22:07:40.776913 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:07:40.776878 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:07:40.916004 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:40.915967 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vppdx_a8f799a1-8427-44a3-a495-74720e82292a/server/0.log" Nov 03 22:07:41.103168 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:41.103140 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2cctc_8116c434-849d-4d12-8d67-9278246bb78b/manager/0.log" Nov 03 22:07:41.320012 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:41.319981 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-95xs7_8c1dc05e-ba05-4715-9236-1eabf21fb915/proxy/0.log" Nov 03 22:07:42.246731 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:42.246701 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-rplcw_f98da404-a032-48b7-8561-92d605ead3c7/revoke-keys/0.log" Nov 03 22:07:42.432207 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:42.432177 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-xkkmg_db307e5d-5d16-4006-a583-56b0988788f4/revoke-keys/0.log" Nov 03 22:07:42.631624 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:42.631546 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-4qvl2_43e22ae4-2e09-4039-ab87-32057416b1ad/revoke-keys/0.log" Nov 03 22:07:42.825937 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:42.825906 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-hzchq_c7500df4-c2fc-47e5-bcbe-330cf017db34/postgres/0.log" Nov 03 22:07:49.675934 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:49.675898 5041 scope.go:122] "RemoveContainer" containerID="af556f47d204d183d907933878f87b6cd4a890f916011432d77e16ef3d2405ac" Nov 03 22:07:55.269623 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:55.269588 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-c446k_b3c86a2a-007c-4a88-87cc-5a97de621e31/manager/0.log" Nov 03 22:07:55.476656 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:55.476621 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6t5ln_f03cef83-f82a-4b4c-901c-e091f6fc58dd/manager/0.log" Nov 03 22:07:55.858740 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:55.858649 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6s2gm_c5b685e7-1863-4500-a5f6-1e58b8335e76/manager/0.log" Nov 03 22:07:56.284838 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:56.284801 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ghgpz_418620a8-7e15-42ac-97fa-ad9568954b5f/manager/0.log" Nov 03 22:07:56.835264 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:56.835230 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs_982671c7-73d5-48b1-9067-d644e1b956d4/istio-proxy/0.log" Nov 03 22:07:57.614309 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:57.614280 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-6kgrf_bd662fa8-196b-4cad-b414-1bc2dbd68aa1/istio-proxy/1.log" Nov 03 22:07:57.822292 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:57.822260 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-8b9jr_69cc2bab-a8e9-4a4a-a114-88663140640d/istio-proxy/0.log" Nov 03 22:07:58.600240 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:58.600210 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-lt4j4_9be23a49-ab91-409d-b131-cb4a917a58e3/istio-proxy/0.log" Nov 03 22:07:58.795143 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:58.795112 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-wc8rh_44bc42a8-430e-44ec-b717-d1a2436968f5/istio-proxy/0.log" Nov 03 22:07:59.589495 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:59.589462 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8_1d77b6f1-de97-4e82-9fb6-0735930827a2/istio-proxy/0.log" Nov 03 22:07:59.775296 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:07:59.775268 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zlxvh_e63ff686-c19d-485e-b503-22cc297aa852/payload-pre-processing/0.log" Nov 03 22:08:00.147466 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:08:00.147436 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9f5c5b5d5-hpzcq_0852ff0f-d078-452f-bb56-3c13512f57ce/router/0.log" Nov 03 22:08:00.856428 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:08:00.856398 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-bqstk_2f3d92ef-2874-44da-90a5-ff115f8c2755/main/0.log" Nov 03 22:08:01.046385 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:08:01.046354 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-v2pwb_fa892a3f-4760-45ff-b00b-0840970540bc/main/0.log" Nov 03 22:08:01.425555 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:08:01.425519 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m7q9r_1d7a34c6-9da5-45de-9373-50b3b18eb7b1/main/0.log" Nov 03 22:08:01.772319 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:08:01.772286 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-tm9db_6fe7207f-2102-4312-b28d-9c91c22b0ddb/main/0.log" Nov 03 22:08:10.777797 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:08:10.777756 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:08:40.776797 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:08:40.776757 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:08:49.703709 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:08:49.703676 5041 scope.go:122] "RemoveContainer" containerID="d710cb0f04a163dd3d55b4f0ca78a1a9818e00b56ef65250a3af7800d3cfdf80" Nov 03 22:08:49.713441 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:08:49.713419 5041 scope.go:122] "RemoveContainer" containerID="3213f878f5da190d412764ea521c0ed0906aade69a25ac7d1fdfd02a9bfcb507" Nov 03 22:09:10.777334 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:09:10.777295 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:09:28.143117 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:09:28.143038 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d575fdb56-jwwm2_89259ba9-9d01-4e60-a875-9b02fb8bd98f/manager/0.log" Nov 03 22:09:28.450038 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:09:28.449938 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d575fdb56-jwwm2_89259ba9-9d01-4e60-a875-9b02fb8bd98f/manager/0.log" Nov 03 22:09:40.776642 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:09:40.776603 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:09:45.860895 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:09:45.860865 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 22:09:45.866561 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:09:45.866535 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 22:09:49.744014 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:09:49.743979 5041 scope.go:122] "RemoveContainer" containerID="94376b313ec4f8c324b44f764a351339d1aa2a245d4b47fde4db7d41782230af" Nov 03 22:10:10.777911 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:10:10.777871 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:10:40.775187 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:10:40.775149 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:10:53.771971 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:10:53.771935 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d575fdb56-jwwm2_89259ba9-9d01-4e60-a875-9b02fb8bd98f/manager/0.log" Nov 03 22:10:54.061765 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:10:54.061639 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d575fdb56-jwwm2_89259ba9-9d01-4e60-a875-9b02fb8bd98f/manager/0.log" Nov 03 22:10:59.910734 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:10:59.910629 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z5jjd_58ec685b-55f0-42a2-9112-6fe8a813fd8a/global-pull-secret-syncer/0.log" Nov 03 22:11:00.025712 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:00.025673 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2"] Nov 03 22:11:00.039251 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:00.039217 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4qvl2"] Nov 03 22:11:00.079054 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:00.079025 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7f6xj_d8a307b9-8e43-4b73-941d-d5e2c035da14/connection-checker/0.log" Nov 03 22:11:00.198168 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:00.198088 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-mq2cr_a734caac-db4c-4f6e-8dd0-415843ecd1b8/connection-checker/0.log" Nov 03 22:11:00.321308 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:00.321279 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-pp685_efc25e42-1fb2-4fb4-b991-ce83b2d174d1/connection-checker/0.log" Nov 03 22:11:00.621252 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:00.621218 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e22ae4-2e09-4039-ab87-32057416b1ad" path="/var/lib/kubelet/pods/43e22ae4-2e09-4039-ab87-32057416b1ad/volumes" Nov 03 22:11:00.776871 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:00.776836 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vh4hd_1f738a71-6d2b-4ade-ac92-990cf0a2b311/konnectivity-agent/0.log" Nov 03 22:11:00.884564 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:00.884481 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-253.ec2.internal_9dff6629a956b5c1c96ca015020c4e26/haproxy/0.log" Nov 03 22:11:07.931160 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:07.931115 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-c446k_b3c86a2a-007c-4a88-87cc-5a97de621e31/manager/0.log" Nov 03 22:11:08.050578 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:08.050550 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6t5ln_f03cef83-f82a-4b4c-901c-e091f6fc58dd/manager/0.log" Nov 03 22:11:08.311015 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:08.310980 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6s2gm_c5b685e7-1863-4500-a5f6-1e58b8335e76/manager/0.log" Nov 03 22:11:08.561153 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:08.561048 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ghgpz_418620a8-7e15-42ac-97fa-ad9568954b5f/manager/0.log" Nov 03 22:11:09.628463 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.628423 5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw"] Nov 03 22:11:09.628970 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.628955 5041 cpu_manager.go:414] "containerMap: removing container" podUID="db307e5d-5d16-4006-a583-56b0988788f4" containerName="revoke-keys" Nov 03 22:11:09.629015 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.628971 5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db307e5d-5d16-4006-a583-56b0988788f4" containerName="revoke-keys" Nov 03 22:11:09.629068 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.629058 5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="db307e5d-5d16-4006-a583-56b0988788f4" containerName="revoke-keys" Nov 03 22:11:09.632513 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.632496 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:09.655110 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.655080 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw"] Nov 03 22:11:09.665052 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.665028 5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sl2kc\"/\"default-dockercfg-8mbwx\"" Nov 03 22:11:09.684439 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.684408 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3466036-0bf0-4ce2-a9eb-ba5d3fddd735-proc\") pod \"perf-node-gather-daemonset-4lxkw\" (UID: \"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735\") " pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:09.684632 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.684532 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3466036-0bf0-4ce2-a9eb-ba5d3fddd735-podres\") pod \"perf-node-gather-daemonset-4lxkw\" (UID: \"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735\") " pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:09.684753 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.684638 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3466036-0bf0-4ce2-a9eb-ba5d3fddd735-sys\") pod \"perf-node-gather-daemonset-4lxkw\" (UID: \"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735\") " pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:09.684753 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.684742 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3466036-0bf0-4ce2-a9eb-ba5d3fddd735-lib-modules\") pod \"perf-node-gather-daemonset-4lxkw\" (UID: \"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735\") " pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:09.684875 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.684831 5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l4l5\" (UniqueName: \"kubernetes.io/projected/f3466036-0bf0-4ce2-a9eb-ba5d3fddd735-kube-api-access-7l4l5\") pod \"perf-node-gather-daemonset-4lxkw\" (UID: \"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735\") " pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:09.736120 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.736088 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sl2kc\"/\"openshift-service-ca.crt\"" Nov 03 22:11:09.736306 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.736147 5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sl2kc\"/\"kube-root-ca.crt\"" Nov 03 22:11:09.785336 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.785303 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3466036-0bf0-4ce2-a9eb-ba5d3fddd735-lib-modules\") pod \"perf-node-gather-daemonset-4lxkw\" (UID: \"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735\") " pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:09.785543 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.785348 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l4l5\" (UniqueName: \"kubernetes.io/projected/f3466036-0bf0-4ce2-a9eb-ba5d3fddd735-kube-api-access-7l4l5\") pod \"perf-node-gather-daemonset-4lxkw\" (UID: \"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735\") " pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:09.785543 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.785377 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3466036-0bf0-4ce2-a9eb-ba5d3fddd735-proc\") pod \"perf-node-gather-daemonset-4lxkw\" (UID: \"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735\") " pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:09.785543 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.785430 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3466036-0bf0-4ce2-a9eb-ba5d3fddd735-podres\") pod \"perf-node-gather-daemonset-4lxkw\" (UID: \"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735\") " pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:09.785543 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.785455 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3466036-0bf0-4ce2-a9eb-ba5d3fddd735-lib-modules\") pod \"perf-node-gather-daemonset-4lxkw\" (UID: \"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735\") " pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:09.785543 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.785460 5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3466036-0bf0-4ce2-a9eb-ba5d3fddd735-sys\") pod \"perf-node-gather-daemonset-4lxkw\" (UID: \"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735\") " pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:09.785543 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.785509 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3466036-0bf0-4ce2-a9eb-ba5d3fddd735-sys\") pod \"perf-node-gather-daemonset-4lxkw\" (UID: \"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735\") " pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:09.785543 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.785523 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3466036-0bf0-4ce2-a9eb-ba5d3fddd735-proc\") pod \"perf-node-gather-daemonset-4lxkw\" (UID: \"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735\") " pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:09.785543 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.785536 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3466036-0bf0-4ce2-a9eb-ba5d3fddd735-podres\") pod \"perf-node-gather-daemonset-4lxkw\" (UID: \"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735\") " pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:09.805510 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.805480 5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l4l5\" (UniqueName: \"kubernetes.io/projected/f3466036-0bf0-4ce2-a9eb-ba5d3fddd735-kube-api-access-7l4l5\") pod \"perf-node-gather-daemonset-4lxkw\" (UID: \"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735\") " pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:09.944972 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:09.944881 5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:10.240968 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:10.240938 5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw"] Nov 03 22:11:10.241092 ip-10-0-131-253 kubenswrapper[5041]: W1103 22:11:10.240943 5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf3466036_0bf0_4ce2_a9eb_ba5d3fddd735.slice/crio-adea586f4f396c21c95c0bd5d6fb02c560e657c0c01cc21bf54103c62a1d6427 WatchSource:0}: Error finding container adea586f4f396c21c95c0bd5d6fb02c560e657c0c01cc21bf54103c62a1d6427: Status 404 returned error can't find the container with id adea586f4f396c21c95c0bd5d6fb02c560e657c0c01cc21bf54103c62a1d6427 Nov 03 22:11:10.623233 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:10.623160 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-wq5rr_ddbb8fca-c02a-4e5f-a31f-919194f46f12/cluster-monitoring-operator/0.log" Nov 03 22:11:10.777268 ip-10-0-131-253 kubenswrapper[5041]: E1103 22:11:10.777233 5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-plrfk" containerName="extractor" Nov 03 22:11:10.781187 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:10.781161 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" event={"ID":"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735","Type":"ContainerStarted","Data":"996ae2cfe8b0b5853fe4c86c8c4c9995650ebdd98082199276c5778f911e8dfc"} Nov 03 22:11:10.781326 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:10.781212 5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" event={"ID":"f3466036-0bf0-4ce2-a9eb-ba5d3fddd735","Type":"ContainerStarted","Data":"adea586f4f396c21c95c0bd5d6fb02c560e657c0c01cc21bf54103c62a1d6427"} Nov 03 22:11:10.781326 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:10.781293 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:10.822816 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:10.822763 5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" podStartSLOduration=1.822749615 podStartE2EDuration="1.822749615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:11:10.821018135 +0000 UTC m=+2486.328246416" watchObservedRunningTime="2026-11-03 22:11:10.822749615 +0000 UTC m=+2486.329977895" Nov 03 22:11:12.028226 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:12.028200 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xs5ht_afd2b194-1d73-415c-913d-4d998a51ea4b/node-exporter/0.log" Nov 03 22:11:12.141464 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:12.141438 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xs5ht_afd2b194-1d73-415c-913d-4d998a51ea4b/kube-rbac-proxy/0.log" Nov 03 22:11:12.255056 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:12.255027 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xs5ht_afd2b194-1d73-415c-913d-4d998a51ea4b/init-textfile/0.log" Nov 03 22:11:16.130428 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:16.130384 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-w682j_9205351c-9ed0-4a10-8c52-0124c39a20fe/networking-console-plugin/0.log" Nov 03 22:11:16.798608 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:16.798572 5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sl2kc/perf-node-gather-daemonset-4lxkw" Nov 03 22:11:17.149458 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:17.149379 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/2.log" Nov 03 22:11:17.154041 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:17.154010 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ltzz4_c5c4b6e5-c1f2-4e66-b8c8-6d819d7ee10b/console-operator/3.log" Nov 03 22:11:18.025707 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:18.025671 5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw"] Nov 03 22:11:18.039699 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:18.039646 5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rplcw"] Nov 03 22:11:18.620960 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:18.620927 5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98da404-a032-48b7-8561-92d605ead3c7" path="/var/lib/kubelet/pods/f98da404-a032-48b7-8561-92d605ead3c7/volumes" Nov 03 22:11:19.316301 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:19.316269 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-zthx8_cb8cd0fc-653b-4f07-a501-f121ec73b7d5/volume-data-source-validator/0.log" Nov 03 22:11:21.655342 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:21.655316 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7t4b8_2a8bade0-e7b8-4d61-b71f-fdb4b2517320/dns/0.log" Nov 03 22:11:21.768922 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:21.768893 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7t4b8_2a8bade0-e7b8-4d61-b71f-fdb4b2517320/kube-rbac-proxy/0.log" Nov 03 22:11:22.134277 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:22.134232 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs_982671c7-73d5-48b1-9067-d644e1b956d4/istio-proxy/0.log" Nov 03 22:11:22.246090 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:22.246063 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5wq87_d65ba521-c50d-47b4-82d9-1910dde4fbb1/dns-node-resolver/0.log" Nov 03 22:11:23.929913 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:23.929881 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rwd6w_fb46e9cd-a477-4380-8d82-63a93d0f4099/node-ca/0.log" Nov 03 22:11:24.615939 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:24.615907 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-6kgrf_bd662fa8-196b-4cad-b414-1bc2dbd68aa1/istio-proxy/0.log" Nov 03 22:11:24.617352 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:24.617326 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-6kgrf_bd662fa8-196b-4cad-b414-1bc2dbd68aa1/istio-proxy/1.log" Nov 03 22:11:24.668642 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:24.668609 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-8b9jr_69cc2bab-a8e9-4a4a-a114-88663140640d/istio-proxy/0.log" Nov 03 22:11:24.979322 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:24.979288 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-lt4j4_9be23a49-ab91-409d-b131-cb4a917a58e3/istio-proxy/0.log" Nov 03 22:11:25.056585 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:25.056556 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-wc8rh_44bc42a8-430e-44ec-b717-d1a2436968f5/istio-proxy/0.log" Nov 03 22:11:25.287401 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:25.287320 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cf4665f7f-nlcm8_1d77b6f1-de97-4e82-9fb6-0735930827a2/istio-proxy/0.log" Nov 03 22:11:25.343575 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:25.343547 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zlxvh_e63ff686-c19d-485e-b503-22cc297aa852/payload-pre-processing/0.log" Nov 03 22:11:25.462686 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:25.462636 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9f5c5b5d5-hpzcq_0852ff0f-d078-452f-bb56-3c13512f57ce/router/0.log" Nov 03 22:11:25.795494 ip-10-0-131-253 kubenswrapper[5041]: I1103 22:11:25.795462 5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7c8gchs_982671c7-73d5-48b1-9067-d644e1b956d4/istio-proxy/0.log"