Sep 11 14:14:46.074752 ip-10-0-131-110 systemd[1]: Starting Kubernetes Kubelet... Sep 11 14:14:46.503056 ip-10-0-131-110 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 14:14:46.503056 ip-10-0-131-110 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 14:14:46.503056 ip-10-0-131-110 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 14:14:46.503056 ip-10-0-131-110 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 14:14:46.503056 ip-10-0-131-110 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 14:14:46.504821 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.504738 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 14:14:46.513155 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513137 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:46.513155 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513152 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:46.513155 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513156 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513159 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513162 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513165 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513168 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513172 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513180 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513183 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513186 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513189 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513192 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513195 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513198 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513200 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513203 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513206 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513209 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513212 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513215 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:46.513240 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513218 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513220 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513223 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513225 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513229 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513232 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513234 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513237 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513239 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513242 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513245 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513271 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513274 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513277 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513280 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513282 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513285 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513287 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513290 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513293 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:46.513834 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513296 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513298 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513301 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513303 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513306 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513308 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513311 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513313 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513317 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513321 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513324 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513326 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513329 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513332 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513335 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513339 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513342 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513345 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513348 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:46.514658 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513350 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513353 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513356 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513358 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513361 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513364 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513366 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513369 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513371 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513374 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513376 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513379 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513381 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513384 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513387 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513389 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513392 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513394 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513397 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513399 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513402 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:46.515186 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513404 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513407 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513409 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513412 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513414 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513417 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513421 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513423 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513426 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513429 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513431 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513434 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513437 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513441 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513444 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513446 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513449 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:46.515714 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.513451 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515734 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515745 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515748 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515751 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515754 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515757 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515759 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515763 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515765 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515768 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515771 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515774 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515776 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515779 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515781 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515783 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515786 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515788 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515791 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:46.516130 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515794 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515796 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515799 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515801 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515804 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515806 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515809 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515812 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515817 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515821 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515824 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515828 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515831 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515834 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515837 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515839 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515842 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515845 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515847 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:46.516637 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515850 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515852 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515855 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515857 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515861 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515863 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515866 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515868 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515870 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515873 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515875 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515878 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515880 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515883 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515886 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515888 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515891 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515893 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515896 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515898 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:46.517182 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515901 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515903 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515907 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515909 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515912 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515917 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515921 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515934 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515937 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515940 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515943 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515946 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515949 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515951 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515954 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515957 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515961 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515964 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515968 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:46.517688 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515970 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515973 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515976 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515978 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515981 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515983 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515987 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515990 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515992 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515995 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.515997 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516000 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516003 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516005 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516008 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516011 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516014 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516017 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516020 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:46.518185 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516023 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516027 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516029 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516402 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516407 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516410 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516413 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516415 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516418 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516422 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516427 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516430 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516433 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516436 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516439 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516442 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516445 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516447 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516450 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516454 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:46.518671 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516472 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516476 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516480 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516484 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516487 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516490 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516494 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516496 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516499 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516501 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516504 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516508 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516510 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516513 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516516 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516518 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516521 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516523 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516526 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516528 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:46.519150 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516531 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516533 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516536 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516538 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516541 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516543 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516546 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516548 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516551 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516553 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516556 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516559 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516562 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516564 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516567 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516569 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516572 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516575 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516577 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:46.519679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516579 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516582 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516585 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516587 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516591 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516594 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516597 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516599 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516602 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516605 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516607 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516610 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516612 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516615 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516617 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516620 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516622 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516625 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516628 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516631 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:46.520145 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516633 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516636 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516638 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516641 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516644 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516646 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516649 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516652 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516656 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516659 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516661 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516664 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516666 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516669 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516671 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516674 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516677 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516681 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516684 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516687 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:46.520634 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516689 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516692 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.516695 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516768 2573 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516809 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516814 2573 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516818 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516823 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516826 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516830 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516834 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516839 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516842 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516845 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516849 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516852 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516855 2573 flags.go:64] FLAG: --cgroup-root="" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516858 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516860 2573 flags.go:64] FLAG: --client-ca-file="" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516865 2573 flags.go:64] FLAG: --cloud-provider="external" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516869 2573 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516874 2573 flags.go:64] FLAG: --cluster-domain="" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516876 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516880 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 14:14:46.521143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516883 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516886 2573 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516890 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516892 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516896 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516899 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516902 2573 flags.go:64] FLAG: --contention-profiling="false" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516905 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516908 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516911 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516914 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516917 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516921 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516923 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516926 2573 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516929 2573 flags.go:64] FLAG: --enable-server="true" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516932 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516936 2573 flags.go:64] FLAG: --event-burst="100" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516939 2573 flags.go:64] FLAG: --event-qps="50" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516944 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516947 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516949 2573 flags.go:64] FLAG: --eviction-hard="" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516953 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516956 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516959 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 14:14:46.521749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516962 2573 flags.go:64] FLAG: --eviction-soft="" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516964 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516968 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516971 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516974 2573 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516977 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516980 2573 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516983 2573 flags.go:64] FLAG: --feature-gates="" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516987 2573 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516990 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516993 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516996 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.516999 2573 flags.go:64] FLAG: --healthz-port="10248" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517002 2573 flags.go:64] FLAG: --help="false" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517005 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-131-110.ec2.internal" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517008 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517011 2573 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517013 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517017 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517020 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517023 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517026 2573 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517029 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 14:14:46.522338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517032 2573 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517034 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517038 2573 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517042 2573 flags.go:64] FLAG: --kube-reserved="" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517045 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517048 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517051 2573 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517053 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517056 2573 flags.go:64] FLAG: --lock-file="" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517059 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517062 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517065 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517071 2573 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517074 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517077 2573 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517080 2573 flags.go:64] FLAG: --logging-format="text" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517083 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517086 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517089 2573 flags.go:64] FLAG: --manifest-url="" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517092 2573 flags.go:64] FLAG: --manifest-url-header="" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517096 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517099 2573 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517103 2573 flags.go:64] FLAG: --max-pods="110" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517106 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517109 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 14:14:46.522945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517112 2573 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517115 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517118 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517121 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517124 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517131 2573 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517134 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517137 2573 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517140 2573 flags.go:64] FLAG: --pod-cidr="" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517143 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517148 2573 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517153 2573 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517156 2573 flags.go:64] FLAG: --pods-per-core="0" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517159 2573 flags.go:64] FLAG: --port="10250" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517162 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517165 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0456a99f3dbdc6cee" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517168 2573 flags.go:64] FLAG: --qos-reserved="" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517171 2573 flags.go:64] FLAG: --read-only-port="10255" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517174 2573 flags.go:64] FLAG: --register-node="true" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517177 2573 flags.go:64] FLAG: --register-with-taints="" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517180 2573 flags.go:64] FLAG: --registry-burst="10" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517184 2573 flags.go:64] FLAG: --registry-qps="5" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517187 2573 flags.go:64] FLAG: --reserved-cpus="" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517190 2573 flags.go:64] FLAG: --reserved-memory="" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517193 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 14:14:46.523575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517196 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517200 2573 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517203 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517205 2573 flags.go:64] FLAG: --runonce="false" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517208 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517211 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517214 2573 flags.go:64] FLAG: --seccomp-default="false" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517217 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517220 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517223 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517226 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517229 2573 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517232 2573 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517235 2573 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517237 2573 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517240 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517243 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517246 2573 flags.go:64] FLAG: --system-cgroups="" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517250 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517256 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517258 2573 flags.go:64] FLAG: --tls-cert-file="" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517261 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517265 2573 flags.go:64] FLAG: --tls-min-version="" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517268 2573 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517272 2573 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 14:14:46.524212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517275 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517278 2573 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517281 2573 flags.go:64] FLAG: --v="2" Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517284 2573 flags.go:64] FLAG: --version="false" Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517289 2573 flags.go:64] FLAG: --vmodule="" Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517292 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.517295 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517409 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517412 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517415 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517419 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517422 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517426 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517429 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517432 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517435 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517437 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517440 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517443 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517445 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517448 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:46.524831 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517450 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517453 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517471 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517476 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517484 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517487 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517491 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517494 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517496 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517499 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517503 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517506 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517509 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517511 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517514 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517516 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517520 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517522 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517525 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517528 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:46.525348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517530 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517533 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517536 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517538 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517541 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517544 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517546 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517549 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517551 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517554 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517556 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517559 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517561 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517564 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517566 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517569 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517573 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517576 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517579 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517581 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:46.525861 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517584 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517587 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517590 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517593 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517596 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517598 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517601 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517604 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517606 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517609 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517612 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517614 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517617 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517620 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517622 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517625 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517629 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517633 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517636 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:46.526348 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517638 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517641 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517644 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517646 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517649 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517653 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517656 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517659 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517662 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517666 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517669 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517672 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517674 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517677 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517679 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517683 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517686 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517689 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517692 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:46.526817 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517694 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:46.527289 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517697 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:46.527289 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517700 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:46.527289 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517703 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:46.527289 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517706 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:46.527289 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517709 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:46.527289 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.517711 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:46.527289 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.518422 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 14:14:46.527289 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.526052 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 14:14:46.527289 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.526066 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 14:14:46.527289 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.526085 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 14:14:46.527289 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.526091 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 14:14:46.527289 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526144 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:46.527289 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526148 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:46.527289 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526152 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:46.527289 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526155 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526158 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526161 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526164 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526166 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526169 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526172 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526174 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526177 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526180 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526183 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526186 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526188 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526191 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526194 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526196 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526199 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526201 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526204 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526207 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:46.527724 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526210 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526212 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526215 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526217 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526220 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526222 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526227 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526230 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526232 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526235 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526238 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526241 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526243 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526246 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526248 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526250 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526253 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526256 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526258 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526261 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:46.528220 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526263 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526266 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526268 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526271 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526273 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526276 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526278 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526280 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526283 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526285 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526289 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526293 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526296 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526299 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526303 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526307 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526310 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526313 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526316 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:46.528726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526320 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526322 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526325 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526328 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526330 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526333 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526336 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526338 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526341 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526344 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526346 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526349 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526351 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526354 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526356 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526359 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526362 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526364 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526367 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526369 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:46.529196 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526372 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:46.529704 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526375 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:46.529704 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526377 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:46.529704 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526380 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:46.529704 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526382 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:46.529704 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526385 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:46.529704 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526387 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:46.529704 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526391 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:46.529704 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526395 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:46.529704 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526398 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:46.529704 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526401 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:46.529704 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526404 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:46.529704 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526407 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:46.529704 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526411 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:46.529704 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526414 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:46.529704 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526417 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:46.529704 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526420 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:46.530099 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.526425 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 14:14:46.530099 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526553 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:46.530099 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526559 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:46.530099 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526562 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:46.530099 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526565 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:46.530099 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526568 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:46.530099 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526571 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:46.530099 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526574 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:46.530099 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526576 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:46.530099 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526579 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:46.530099 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526581 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:46.530099 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526584 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:46.530099 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526587 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:46.530099 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526589 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526592 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526595 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526597 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526600 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526603 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526605 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526608 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526610 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526613 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526615 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526618 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526620 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526623 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526626 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526629 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526632 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526635 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526637 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526640 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:46.530454 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526643 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526645 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526648 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526651 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526653 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526656 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526658 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526661 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526663 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526666 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526668 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526671 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526673 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526676 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526679 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526681 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526683 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526686 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526689 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526691 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:46.530982 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526694 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526696 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526699 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526702 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526705 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526707 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526710 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526713 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526715 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526718 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526721 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526723 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526726 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526730 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526734 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526737 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526739 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526742 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526744 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526747 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:46.531482 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526749 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526752 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526755 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526757 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526760 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526762 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526765 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526768 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526770 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526773 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526776 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526778 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526781 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526783 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526786 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526788 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526792 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526794 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526799 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:46.531966 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526804 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:46.532420 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526807 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:46.532420 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526810 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:46.532420 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526814 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:46.532420 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526816 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:46.532420 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526819 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:46.532420 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526823 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:46.532420 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:46.526826 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:46.532420 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.526831 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 14:14:46.532420 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.526947 2573 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 14:14:46.532420 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.530168 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 14:14:46.532420 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.531235 2573 server.go:1019] "Starting client certificate rotation" Sep 11 14:14:46.532420 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.531335 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 14:14:46.532420 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.531373 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 14:14:46.534053 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.534040 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 14:14:46.535189 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.535172 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 14:14:46.546052 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.546031 2573 log.go:25] "Validated CRI v1 runtime API" Sep 11 14:14:46.549242 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.549226 2573 log.go:25] "Validated CRI v1 image API" Sep 11 14:14:46.550088 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.550074 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 14:14:46.558075 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.558051 2573 fs.go:135] Filesystem UUIDs: map[47b31803-e242-43de-adb3-049cd3dadebf:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a97eba06-4719-4639-b6ec-b4bb6e0f68bd:/dev/nvme0n1p4] Sep 11 14:14:46.558075 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.558068 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 14:14:46.563393 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.563289 2573 manager.go:216] Machine: {Timestamp:2026-09-11 14:14:46.561865121 +0000 UTC m=+0.382206933 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3115210 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26796e43610bfdc69048f7030d02f3 SystemUUID:ec26796e-4361-0bfd-c690-48f7030d02f3 BootID:2e5388a8-e951-41cc-855e-06ec04fedb43 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:22:8b:e8:34:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:22:8b:e8:34:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:34:fb:ab:d9:de Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 14:14:46.563393 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.563390 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 14:14:46.563524 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.563509 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 14:14:46.565052 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.565035 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 14:14:46.566087 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.566061 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 14:14:46.566212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.566089 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-110.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 14:14:46.567649 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.567638 2573 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 14:14:46.567684 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.567652 2573 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 14:14:46.567684 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.567660 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 14:14:46.568485 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.568473 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 14:14:46.569908 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.569899 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 14:14:46.574192 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.574181 2573 state_mem.go:36] "Initialized new in-memory state store" Sep 11 14:14:46.574298 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.574289 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 14:14:46.574327 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.574321 2573 kubelet.go:486] "Attempting to sync node with API server" Sep 11 14:14:46.574361 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.574330 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 14:14:46.574361 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.574344 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 14:14:46.574361 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.574353 2573 kubelet.go:398] "Adding apiserver pod source" Sep 11 14:14:46.574361 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.574361 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 14:14:46.574720 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.574709 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 14:14:46.574753 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.574728 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 14:14:46.576901 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.576887 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 14:14:46.579649 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.579633 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 14:14:46.579723 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.579672 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 14:14:46.581710 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.581695 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 14:14:46.581793 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.581715 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 14:14:46.581793 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.581724 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 14:14:46.581793 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.581731 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 14:14:46.581793 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.581743 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 14:14:46.581793 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.581751 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 14:14:46.581793 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.581770 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 14:14:46.581793 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.581778 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 14:14:46.581793 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.581787 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 14:14:46.581793 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.581795 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 14:14:46.582065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.581808 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 14:14:46.582065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.581816 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 14:14:46.582692 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.582681 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 14:14:46.582752 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.582695 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 14:14:46.586471 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.586442 2573 server.go:1268] "Started kubelet" Sep 11 14:14:46.586539 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.586515 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 14:14:46.586652 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.586602 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 14:14:46.586693 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.586676 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 14:14:46.587198 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.587179 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 14:14:46.587268 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.587192 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 14:14:46.587718 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.587703 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-110.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 14:14:46.588216 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.588199 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 14:14:46.588584 ip-10-0-131-110 systemd[1]: Started Kubernetes Kubelet. Sep 11 14:14:46.594918 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.594895 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 14:14:46.595099 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.595086 2573 server.go:310] "Adding debug handlers to kubelet server" Sep 11 14:14:46.596101 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.594988 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-110.ec2.internal.18d449ca9fd0dabc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-110.ec2.internal,UID:ip-10-0-131-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-110.ec2.internal,},FirstTimestamp:2026-09-11 14:14:46.586415804 +0000 UTC m=+0.406757616,LastTimestamp:2026-09-11 14:14:46.586415804 +0000 UTC m=+0.406757616,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-110.ec2.internal,}" Sep 11 14:14:46.599162 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.599151 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 14:14:46.599229 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.599171 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 14:14:46.599284 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.599241 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-110.ec2.internal\" not found" Sep 11 14:14:46.599870 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.599847 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 14:14:46.599870 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.599849 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 14:14:46.599870 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.599872 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 14:14:46.600059 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.599959 2573 reconstruct.go:97] "Volume reconstruction finished" Sep 11 14:14:46.600059 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.599967 2573 reconciler.go:29] "Reconciler: start to sync state" Sep 11 14:14:46.600750 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.600733 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 14:14:46.600750 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.600750 2573 factory.go:55] Registering systemd factory Sep 11 14:14:46.600864 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.600756 2573 factory.go:223] Registration of the systemd container factory successfully Sep 11 14:14:46.600947 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.600933 2573 factory.go:153] Registering CRI-O factory Sep 11 14:14:46.600947 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.600947 2573 factory.go:223] Registration of the crio container factory successfully Sep 11 14:14:46.601008 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.600961 2573 factory.go:103] Registering Raw factory Sep 11 14:14:46.601008 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.600973 2573 manager.go:1196] Started watching for new ooms in manager Sep 11 14:14:46.601313 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.601291 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 14:14:46.601368 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.601304 2573 manager.go:349] Starting recovery of all containers Sep 11 14:14:46.601368 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.601348 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 14:14:46.611183 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.611166 2573 manager.go:354] Recovery completed Sep 11 14:14:46.615104 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.615091 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:46.616957 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.616930 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:46.617047 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.616967 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:46.617047 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.616978 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:46.617586 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.617569 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:46.617671 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.617558 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 14:14:46.617671 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.617636 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 14:14:46.617671 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.617656 2573 state_mem.go:36] "Initialized new in-memory state store" Sep 11 14:14:46.618346 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.618331 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:46.618424 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.618354 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:46.618424 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.618363 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:46.619030 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.619016 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6b57d" Sep 11 14:14:46.619359 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.619348 2573 policy_none.go:49] "None policy: Start" Sep 11 14:14:46.619391 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.619363 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 14:14:46.619391 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.619374 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 14:14:46.621104 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.621094 2573 policy_none.go:47] "Start" Sep 11 14:14:46.627673 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.627652 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6b57d" Sep 11 14:14:46.629301 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.629109 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-110.ec2.internal.18d449caa1a2d478 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-110.ec2.internal,UID:ip-10-0-131-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-110.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-110.ec2.internal,},FirstTimestamp:2026-09-11 14:14:46.616953976 +0000 UTC m=+0.437295787,LastTimestamp:2026-09-11 14:14:46.616953976 +0000 UTC m=+0.437295787,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-110.ec2.internal,}" Sep 11 14:14:46.631850 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.631725 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-110.ec2.internal.18d449caa1a31be3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-110.ec2.internal,UID:ip-10-0-131-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-110.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-110.ec2.internal,},FirstTimestamp:2026-09-11 14:14:46.616972259 +0000 UTC m=+0.437314071,LastTimestamp:2026-09-11 14:14:46.616972259 +0000 UTC m=+0.437314071,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-110.ec2.internal,}" Sep 11 14:14:46.646648 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.646635 2573 manager.go:337] "Starting Device Plugin manager" Sep 11 14:14:46.676127 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.646672 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 14:14:46.676127 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.646684 2573 server.go:87] "Starting device plugin registration server" Sep 11 14:14:46.676127 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.646890 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 14:14:46.676127 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.646899 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 14:14:46.676127 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.647096 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 14:14:46.676127 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.647156 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 14:14:46.676127 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.647164 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 14:14:46.676127 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.647537 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 14:14:46.676127 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.647579 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-110.ec2.internal\" not found" Sep 11 14:14:46.695768 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.695746 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 14:14:46.696890 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.696871 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 14:14:46.696963 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.696919 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 14:14:46.696963 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.696942 2573 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 14:14:46.697071 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.696982 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 14:14:46.699580 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.699557 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:46.747798 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.747773 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:46.748806 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.748786 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:46.748879 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.748816 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:46.748879 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.748827 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:46.748879 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.748848 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-110.ec2.internal" Sep 11 14:14:46.758476 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.758427 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-110.ec2.internal" Sep 11 14:14:46.758476 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.758447 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-110.ec2.internal\": node \"ip-10-0-131-110.ec2.internal\" not found" Sep 11 14:14:46.797717 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.797693 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-110.ec2.internal"] Sep 11 14:14:46.797789 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.797778 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:46.798620 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.798602 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:46.798699 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.798636 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:46.798699 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.798647 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:46.798775 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.798735 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-110.ec2.internal\" not found" Sep 11 14:14:46.800773 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.800760 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:46.800892 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.800880 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal" Sep 11 14:14:46.800933 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.800907 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:46.801469 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.801436 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:46.801469 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.801448 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:46.801597 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.801479 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:46.801597 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.801482 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:46.801597 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.801511 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:46.801597 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.801491 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:46.803674 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.803656 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.ec2.internal" Sep 11 14:14:46.803762 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.803692 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:46.804367 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.804351 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:46.804445 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.804377 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:46.804445 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.804388 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:46.827996 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.827974 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-110.ec2.internal\" not found" node="ip-10-0-131-110.ec2.internal" Sep 11 14:14:46.831367 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.831353 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-110.ec2.internal\" not found" node="ip-10-0-131-110.ec2.internal" Sep 11 14:14:46.899800 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.899779 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-110.ec2.internal\" not found" Sep 11 14:14:46.900921 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.900903 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/344072aebe26cd1a15ff23a2d0a3fde8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal\" (UID: \"344072aebe26cd1a15ff23a2d0a3fde8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal" Sep 11 14:14:46.900993 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.900933 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ecb7720b7510a32693de89b52bd85266-config\") pod \"kube-apiserver-proxy-ip-10-0-131-110.ec2.internal\" (UID: \"ecb7720b7510a32693de89b52bd85266\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.ec2.internal" Sep 11 14:14:46.900993 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:46.900968 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/344072aebe26cd1a15ff23a2d0a3fde8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal\" (UID: \"344072aebe26cd1a15ff23a2d0a3fde8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal" Sep 11 14:14:46.999851 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:46.999824 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-110.ec2.internal\" not found" Sep 11 14:14:47.002026 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:47.002010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/344072aebe26cd1a15ff23a2d0a3fde8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal\" (UID: \"344072aebe26cd1a15ff23a2d0a3fde8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal" Sep 11 14:14:47.002072 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:47.002035 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/344072aebe26cd1a15ff23a2d0a3fde8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal\" (UID: \"344072aebe26cd1a15ff23a2d0a3fde8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal" Sep 11 14:14:47.002072 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:47.002053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ecb7720b7510a32693de89b52bd85266-config\") pod \"kube-apiserver-proxy-ip-10-0-131-110.ec2.internal\" (UID: \"ecb7720b7510a32693de89b52bd85266\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.ec2.internal" Sep 11 14:14:47.002146 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:47.002102 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ecb7720b7510a32693de89b52bd85266-config\") pod \"kube-apiserver-proxy-ip-10-0-131-110.ec2.internal\" (UID: \"ecb7720b7510a32693de89b52bd85266\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.ec2.internal" Sep 11 14:14:47.002146 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:47.002109 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/344072aebe26cd1a15ff23a2d0a3fde8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal\" (UID: \"344072aebe26cd1a15ff23a2d0a3fde8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal" Sep 11 14:14:47.002146 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:47.002130 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/344072aebe26cd1a15ff23a2d0a3fde8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal\" (UID: \"344072aebe26cd1a15ff23a2d0a3fde8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal" Sep 11 14:14:47.100491 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:47.100420 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-110.ec2.internal\" not found" Sep 11 14:14:47.130710 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:47.130694 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal" Sep 11 14:14:47.133622 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:47.133596 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.ec2.internal" Sep 11 14:14:47.201292 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:47.201267 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-110.ec2.internal\" not found" Sep 11 14:14:47.301767 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:47.301742 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-110.ec2.internal\" not found" Sep 11 14:14:48.885663 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:48.885593 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-110.ec2.internal\" not found" Sep 11 14:14:48.986183 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:48.986145 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-110.ec2.internal\" not found" Sep 11 14:14:49.014456 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.014435 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 14:14:49.014851 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.014574 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 14:14:49.086767 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:49.086744 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-110.ec2.internal\" not found" Sep 11 14:14:49.117975 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.117938 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 14:09:48 +0000 UTC" deadline="2028-06-14 23:26:10.136276728 +0000 UTC" Sep 11 14:14:49.117975 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.117966 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15417h11m21.018313621s" Sep 11 14:14:49.135417 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.135396 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:49.187324 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:49.187266 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-110.ec2.internal\" not found" Sep 11 14:14:49.271590 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.271556 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:49.272630 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.272613 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:49.272721 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.272644 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:49.272721 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.272654 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:49.287657 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:49.287633 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-110.ec2.internal\" not found" Sep 11 14:14:49.346028 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:49.346000 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344072aebe26cd1a15ff23a2d0a3fde8.slice/crio-893a4c80149a0a4dba87fda62a099636546d5d95a32840079dc7bb2c5c9e2b79 WatchSource:0}: Error finding container 893a4c80149a0a4dba87fda62a099636546d5d95a32840079dc7bb2c5c9e2b79: Status 404 returned error can't find the container with id 893a4c80149a0a4dba87fda62a099636546d5d95a32840079dc7bb2c5c9e2b79 Sep 11 14:14:49.346259 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:49.346240 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb7720b7510a32693de89b52bd85266.slice/crio-87a064f56b2bcb3f936ab115ac75dcb41f071c4c414a3fe29dc336385ed828ca WatchSource:0}: Error finding container 87a064f56b2bcb3f936ab115ac75dcb41f071c4c414a3fe29dc336385ed828ca: Status 404 returned error can't find the container with id 87a064f56b2bcb3f936ab115ac75dcb41f071c4c414a3fe29dc336385ed828ca Sep 11 14:14:49.351286 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.351270 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:14:49.382265 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.382241 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:49.388036 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:49.388017 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-110.ec2.internal\" not found" Sep 11 14:14:49.488735 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:49.488663 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-110.ec2.internal\" not found" Sep 11 14:14:49.489553 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.489537 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:49.582987 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.582955 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal" Sep 11 14:14:49.605656 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.605634 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 14:14:49.606306 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.606294 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.ec2.internal" Sep 11 14:14:49.629295 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.629275 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 14:14:49.983699 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:49.983618 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:50.058930 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.058903 2573 apiserver.go:52] "Watching apiserver" Sep 11 14:14:50.068202 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.068171 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 14:14:50.069481 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.069073 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-m8lns","kube-system/konnectivity-agent-sn4pw","kube-system/kube-apiserver-proxy-ip-10-0-131-110.ec2.internal","openshift-image-registry/node-ca-pb9cz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal","openshift-multus/multus-additional-cni-plugins-jk42w","openshift-multus/network-metrics-daemon-fh5nz","openshift-network-diagnostics/network-check-target-fh2vt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j","openshift-cluster-node-tuning-operator/tuned-4xt8r","openshift-dns/node-resolver-l7mvd","openshift-multus/multus-cndlm","openshift-network-operator/iptables-alerter-gmprw","openshift-ovn-kubernetes/ovnkube-node-d87x6"] Sep 11 14:14:50.071669 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.071647 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:14:50.071780 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.071742 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:14:50.073810 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.073786 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l7mvd" Sep 11 14:14:50.076375 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.076074 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.076375 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.076237 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z27gd\"" Sep 11 14:14:50.076661 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.076633 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 14:14:50.076748 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.076641 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 14:14:50.078310 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.078279 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 14:14:50.078409 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.078353 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pb9cz" Sep 11 14:14:50.078409 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.078386 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 14:14:50.078873 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.078847 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 14:14:50.079091 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.079073 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8qnb5\"" Sep 11 14:14:50.079174 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.079098 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 14:14:50.079626 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.079606 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 14:14:50.079711 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.079699 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 14:14:50.080805 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.080787 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:14:50.080908 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.080859 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:14:50.080982 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.080963 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 14:14:50.081157 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.081128 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 14:14:50.081272 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.081197 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 14:14:50.081498 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.081482 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hq7h7\"" Sep 11 14:14:50.083050 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.083031 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.083150 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.083137 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 14:14:50.085316 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.085296 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.086621 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.086601 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 14:14:50.086621 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.086615 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 14:14:50.086753 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.086640 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 14:14:50.087017 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.086998 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 14:14:50.087113 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.087076 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 14:14:50.087366 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.087349 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xqtkw\"" Sep 11 14:14:50.087602 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.087584 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gmprw" Sep 11 14:14:50.087689 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.087674 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 14:14:50.088005 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.087986 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g7f7b\"" Sep 11 14:14:50.088209 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.088178 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 14:14:50.088384 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.088372 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 14:14:50.090443 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.089991 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:14:50.090443 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.090114 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sn4pw" Sep 11 14:14:50.090443 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.090379 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qzrvc\"" Sep 11 14:14:50.090737 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.090708 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 14:14:50.090794 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.090745 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 14:14:50.092473 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.092451 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.092713 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.092694 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 14:14:50.092969 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.092953 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bmlww\"" Sep 11 14:14:50.093240 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.093225 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 14:14:50.094334 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094314 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j27cd\" (UniqueName: \"kubernetes.io/projected/9c142fa3-73aa-434a-a3c8-9d3694c03001-kube-api-access-j27cd\") pod \"node-ca-pb9cz\" (UID: \"9c142fa3-73aa-434a-a3c8-9d3694c03001\") " pod="openshift-image-registry/node-ca-pb9cz" Sep 11 14:14:50.094473 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094349 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-etc-selinux\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.094473 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094380 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-run-ovn\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.094473 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094405 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-cni-bin\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.094473 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-run-systemd\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.094473 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094450 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-node-log\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.094737 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094477 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-var-lib-openvswitch\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.094737 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094505 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-kubelet\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.094737 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094530 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eec501bb-6445-4360-8609-7f5a8f557a6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.094737 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094552 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsn77\" (UniqueName: \"kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77\") pod \"network-check-target-fh2vt\" (UID: \"65f1ece3-078e-4c34-9293-e2a4e6007ea8\") " pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:14:50.094737 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094583 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-systemd-units\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.094737 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094600 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eec501bb-6445-4360-8609-7f5a8f557a6a-system-cni-dir\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.094737 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094614 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.094737 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094632 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-registration-dir\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.094737 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094653 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-run-ovn-kubernetes\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.094737 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094670 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c142fa3-73aa-434a-a3c8-9d3694c03001-host\") pod \"node-ca-pb9cz\" (UID: \"9c142fa3-73aa-434a-a3c8-9d3694c03001\") " pod="openshift-image-registry/node-ca-pb9cz" Sep 11 14:14:50.094737 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094692 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-socket-dir\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094745 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-device-dir\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094766 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwwkv\" (UniqueName: \"kubernetes.io/projected/2b65bf7c-bec3-441a-97da-e12835ab7070-kube-api-access-wwwkv\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094782 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zndxk\" (UniqueName: \"kubernetes.io/projected/ee1912ca-8cb0-4013-bc95-5cb2210bbfe3-kube-api-access-zndxk\") pod \"node-resolver-l7mvd\" (UID: \"ee1912ca-8cb0-4013-bc95-5cb2210bbfe3\") " pod="openshift-dns/node-resolver-l7mvd" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094795 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/79da16fc-0e3e-44fe-90be-16a0d3d76de3-ovn-node-metrics-cert\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094808 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tljcm\" (UniqueName: \"kubernetes.io/projected/79da16fc-0e3e-44fe-90be-16a0d3d76de3-kube-api-access-tljcm\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094827 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eec501bb-6445-4360-8609-7f5a8f557a6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094863 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-824l6\" (UniqueName: \"kubernetes.io/projected/eec501bb-6445-4360-8609-7f5a8f557a6a-kube-api-access-824l6\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094885 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-sys-fs\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094931 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-cni-netd\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094954 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/79da16fc-0e3e-44fe-90be-16a0d3d76de3-env-overrides\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.094976 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-run-openvswitch\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095014 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee1912ca-8cb0-4013-bc95-5cb2210bbfe3-hosts-file\") pod \"node-resolver-l7mvd\" (UID: \"ee1912ca-8cb0-4013-bc95-5cb2210bbfe3\") " pod="openshift-dns/node-resolver-l7mvd" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095043 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095043 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1912ca-8cb0-4013-bc95-5cb2210bbfe3-tmp-dir\") pod \"node-resolver-l7mvd\" (UID: \"ee1912ca-8cb0-4013-bc95-5cb2210bbfe3\") " pod="openshift-dns/node-resolver-l7mvd" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095094 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-slash\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095137 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eec501bb-6445-4360-8609-7f5a8f557a6a-cnibin\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.095276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095160 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dwfz2\"" Sep 11 14:14:50.096154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095165 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-run-netns\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.096154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095207 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-etc-openvswitch\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.096154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095239 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-log-socket\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.096154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095259 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c142fa3-73aa-434a-a3c8-9d3694c03001-serviceca\") pod \"node-ca-pb9cz\" (UID: \"9c142fa3-73aa-434a-a3c8-9d3694c03001\") " pod="openshift-image-registry/node-ca-pb9cz" Sep 11 14:14:50.096154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095280 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eec501bb-6445-4360-8609-7f5a8f557a6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.096154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095329 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eec501bb-6445-4360-8609-7f5a8f557a6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.096154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095364 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 14:14:50.096154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095365 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs\") pod \"network-metrics-daemon-fh5nz\" (UID: \"afa6cfac-b37f-4e64-a308-658d8a3b62d9\") " pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:14:50.096154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095441 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmfp9\" (UniqueName: \"kubernetes.io/projected/afa6cfac-b37f-4e64-a308-658d8a3b62d9-kube-api-access-lmfp9\") pod \"network-metrics-daemon-fh5nz\" (UID: \"afa6cfac-b37f-4e64-a308-658d8a3b62d9\") " pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:14:50.096154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095480 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.096154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095527 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/79da16fc-0e3e-44fe-90be-16a0d3d76de3-ovnkube-config\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.096154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095550 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/79da16fc-0e3e-44fe-90be-16a0d3d76de3-ovnkube-script-lib\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.096154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.095600 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:50.096154 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.095651 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:14:50.097325 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.097305 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.099900 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.099878 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 14:14:50.100001 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.099926 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h65r4\"" Sep 11 14:14:50.100142 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.100124 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 14:14:50.124605 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.124583 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7986l" Sep 11 14:14:50.133477 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.133460 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7986l" Sep 11 14:14:50.184264 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.184238 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 14:14:50.186467 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.186421 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.ec2.internal" event={"ID":"ecb7720b7510a32693de89b52bd85266","Type":"ContainerStarted","Data":"87a064f56b2bcb3f936ab115ac75dcb41f071c4c414a3fe29dc336385ed828ca"} Sep 11 14:14:50.188391 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.188339 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal" event={"ID":"344072aebe26cd1a15ff23a2d0a3fde8","Type":"ContainerStarted","Data":"893a4c80149a0a4dba87fda62a099636546d5d95a32840079dc7bb2c5c9e2b79"} Sep 11 14:14:50.195758 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.195725 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-cni-netd\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.195867 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.195759 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/79da16fc-0e3e-44fe-90be-16a0d3d76de3-env-overrides\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.195867 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.195783 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-run-openvswitch\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.195867 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.195806 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee1912ca-8cb0-4013-bc95-5cb2210bbfe3-hosts-file\") pod \"node-resolver-l7mvd\" (UID: \"ee1912ca-8cb0-4013-bc95-5cb2210bbfe3\") " pod="openshift-dns/node-resolver-l7mvd" Sep 11 14:14:50.195867 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.195844 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs\") pod \"network-metrics-daemon-fh5nz\" (UID: \"afa6cfac-b37f-4e64-a308-658d8a3b62d9\") " pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:14:50.196070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.195869 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmfp9\" (UniqueName: \"kubernetes.io/projected/afa6cfac-b37f-4e64-a308-658d8a3b62d9-kube-api-access-lmfp9\") pod \"network-metrics-daemon-fh5nz\" (UID: \"afa6cfac-b37f-4e64-a308-658d8a3b62d9\") " pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:14:50.196070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.195872 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-cni-netd\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.196070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.195897 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-sysconfig\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.196070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.195923 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-system-cni-dir\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.196070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.195939 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee1912ca-8cb0-4013-bc95-5cb2210bbfe3-hosts-file\") pod \"node-resolver-l7mvd\" (UID: \"ee1912ca-8cb0-4013-bc95-5cb2210bbfe3\") " pod="openshift-dns/node-resolver-l7mvd" Sep 11 14:14:50.196070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.195947 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-run-netns\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.196070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.195968 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c142fa3-73aa-434a-a3c8-9d3694c03001-serviceca\") pod \"node-ca-pb9cz\" (UID: \"9c142fa3-73aa-434a-a3c8-9d3694c03001\") " pod="openshift-image-registry/node-ca-pb9cz" Sep 11 14:14:50.196070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.195978 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-run-openvswitch\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.196070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.195991 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmvr9\" (UniqueName: \"kubernetes.io/projected/545efa3b-e0a6-4c10-bd51-6efd7f0a1a18-kube-api-access-xmvr9\") pod \"iptables-alerter-gmprw\" (UID: \"545efa3b-e0a6-4c10-bd51-6efd7f0a1a18\") " pod="openshift-network-operator/iptables-alerter-gmprw" Sep 11 14:14:50.196070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196014 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d87c49f-6095-47cf-8cc2-736f36cb9a74-agent-certs\") pod \"konnectivity-agent-sn4pw\" (UID: \"7d87c49f-6095-47cf-8cc2-736f36cb9a74\") " pod="kube-system/konnectivity-agent-sn4pw" Sep 11 14:14:50.196070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196036 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-lib-modules\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.196070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196065 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3612f9b-fe42-47fb-b60e-81d598756204-etc-tuned\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80eaab0c-0595-443d-84a6-646467d848dd-cni-binary-copy\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196111 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-hostroot\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196136 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-sysctl-conf\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-etc-kubernetes\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196181 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-cni-bin\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196215 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de580cd5-5351-479c-a9ad-0c0936042074-dbus\") pod \"global-pull-secret-syncer-m8lns\" (UID: \"de580cd5-5351-479c-a9ad-0c0936042074\") " pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196237 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-sysctl-d\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196263 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-kubelet\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196291 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eec501bb-6445-4360-8609-7f5a8f557a6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196299 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/79da16fc-0e3e-44fe-90be-16a0d3d76de3-env-overrides\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196316 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsn77\" (UniqueName: \"kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77\") pod \"network-check-target-fh2vt\" (UID: \"65f1ece3-078e-4c34-9293-e2a4e6007ea8\") " pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196339 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/545efa3b-e0a6-4c10-bd51-6efd7f0a1a18-iptables-alerter-script\") pod \"iptables-alerter-gmprw\" (UID: \"545efa3b-e0a6-4c10-bd51-6efd7f0a1a18\") " pod="openshift-network-operator/iptables-alerter-gmprw" Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.196114 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196395 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-host\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196396 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-run-netns\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.196441 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs podName:afa6cfac-b37f-4e64-a308-658d8a3b62d9 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:50.696407302 +0000 UTC m=+3.033375489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs") pod "network-metrics-daemon-fh5nz" (UID: "afa6cfac-b37f-4e64-a308-658d8a3b62d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:50.196641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196464 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdpnd\" (UniqueName: \"kubernetes.io/projected/d3612f9b-fe42-47fb-b60e-81d598756204-kube-api-access-zdpnd\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196528 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-cni-bin\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196585 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-kubelet\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196623 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-systemd-units\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196731 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eec501bb-6445-4360-8609-7f5a8f557a6a-system-cni-dir\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196763 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196785 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c142fa3-73aa-434a-a3c8-9d3694c03001-serviceca\") pod \"node-ca-pb9cz\" (UID: \"9c142fa3-73aa-434a-a3c8-9d3694c03001\") " pod="openshift-image-registry/node-ca-pb9cz" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196822 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-socket-dir\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196854 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196875 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-var-lib-kubelet\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196875 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-systemd-units\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zndxk\" (UniqueName: \"kubernetes.io/projected/ee1912ca-8cb0-4013-bc95-5cb2210bbfe3-kube-api-access-zndxk\") pod \"node-resolver-l7mvd\" (UID: \"ee1912ca-8cb0-4013-bc95-5cb2210bbfe3\") " pod="openshift-dns/node-resolver-l7mvd" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196954 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-socket-dir\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196955 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eec501bb-6445-4360-8609-7f5a8f557a6a-system-cni-dir\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.196988 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/79da16fc-0e3e-44fe-90be-16a0d3d76de3-ovn-node-metrics-cert\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-824l6\" (UniqueName: \"kubernetes.io/projected/eec501bb-6445-4360-8609-7f5a8f557a6a-kube-api-access-824l6\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197069 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eec501bb-6445-4360-8609-7f5a8f557a6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.197460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197189 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-sys-fs\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197341 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-systemd\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197356 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-sys-fs\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197390 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-var-lib-cni-bin\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197429 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-var-lib-cni-multus\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197445 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197456 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-var-lib-kubelet\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197484 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1912ca-8cb0-4013-bc95-5cb2210bbfe3-tmp-dir\") pod \"node-resolver-l7mvd\" (UID: \"ee1912ca-8cb0-4013-bc95-5cb2210bbfe3\") " pod="openshift-dns/node-resolver-l7mvd" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197507 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-slash\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eec501bb-6445-4360-8609-7f5a8f557a6a-cnibin\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197554 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eec501bb-6445-4360-8609-7f5a8f557a6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197604 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-slash\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197626 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eec501bb-6445-4360-8609-7f5a8f557a6a-cnibin\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197643 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d87c49f-6095-47cf-8cc2-736f36cb9a74-konnectivity-ca\") pod \"konnectivity-agent-sn4pw\" (UID: \"7d87c49f-6095-47cf-8cc2-736f36cb9a74\") " pod="kube-system/konnectivity-agent-sn4pw" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197670 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3612f9b-fe42-47fb-b60e-81d598756204-tmp\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197700 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-run-multus-certs\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197727 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-etc-openvswitch\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.198278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197748 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-log-socket\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197770 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eec501bb-6445-4360-8609-7f5a8f557a6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197809 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvxh7\" (UniqueName: \"kubernetes.io/projected/80eaab0c-0595-443d-84a6-646467d848dd-kube-api-access-cvxh7\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197852 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/79da16fc-0e3e-44fe-90be-16a0d3d76de3-ovnkube-config\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197885 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-etc-openvswitch\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197902 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/79da16fc-0e3e-44fe-90be-16a0d3d76de3-ovnkube-script-lib\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197806 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1912ca-8cb0-4013-bc95-5cb2210bbfe3-tmp-dir\") pod \"node-resolver-l7mvd\" (UID: \"ee1912ca-8cb0-4013-bc95-5cb2210bbfe3\") " pod="openshift-dns/node-resolver-l7mvd" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197959 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j27cd\" (UniqueName: \"kubernetes.io/projected/9c142fa3-73aa-434a-a3c8-9d3694c03001-kube-api-access-j27cd\") pod \"node-ca-pb9cz\" (UID: \"9c142fa3-73aa-434a-a3c8-9d3694c03001\") " pod="openshift-image-registry/node-ca-pb9cz" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.197992 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eec501bb-6445-4360-8609-7f5a8f557a6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198071 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-log-socket\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198182 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-etc-selinux\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198215 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-run\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198238 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-run-ovn\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198247 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eec501bb-6445-4360-8609-7f5a8f557a6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198266 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/545efa3b-e0a6-4c10-bd51-6efd7f0a1a18-host-slash\") pod \"iptables-alerter-gmprw\" (UID: \"545efa3b-e0a6-4c10-bd51-6efd7f0a1a18\") " pod="openshift-network-operator/iptables-alerter-gmprw" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198282 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-run-systemd\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.199110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198303 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-node-log\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198341 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-var-lib-openvswitch\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198344 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-etc-selinux\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198362 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-sys\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198381 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-run-ovn\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198387 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-cnibin\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198405 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-multus-conf-dir\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198422 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/79da16fc-0e3e-44fe-90be-16a0d3d76de3-ovnkube-script-lib\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198438 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-registration-dir\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198458 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-kubernetes\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198469 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198478 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-multus-cni-dir\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198492 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-run-k8s-cni-cncf-io\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198500 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/79da16fc-0e3e-44fe-90be-16a0d3d76de3-ovnkube-config\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198507 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/80eaab0c-0595-443d-84a6-646467d848dd-multus-daemon-config\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198528 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-node-log\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198544 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-run-ovn-kubernetes\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.199926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198553 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-registration-dir\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.200790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198590 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-var-lib-openvswitch\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.200790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198604 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c142fa3-73aa-434a-a3c8-9d3694c03001-host\") pod \"node-ca-pb9cz\" (UID: \"9c142fa3-73aa-434a-a3c8-9d3694c03001\") " pod="openshift-image-registry/node-ca-pb9cz" Sep 11 14:14:50.200790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198618 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-host-run-ovn-kubernetes\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.200790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198627 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-device-dir\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.200790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198639 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c142fa3-73aa-434a-a3c8-9d3694c03001-host\") pod \"node-ca-pb9cz\" (UID: \"9c142fa3-73aa-434a-a3c8-9d3694c03001\") " pod="openshift-image-registry/node-ca-pb9cz" Sep 11 14:14:50.200790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwwkv\" (UniqueName: \"kubernetes.io/projected/2b65bf7c-bec3-441a-97da-e12835ab7070-kube-api-access-wwwkv\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.200790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198657 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/79da16fc-0e3e-44fe-90be-16a0d3d76de3-run-systemd\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.200790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198682 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret\") pod \"global-pull-secret-syncer-m8lns\" (UID: \"de580cd5-5351-479c-a9ad-0c0936042074\") " pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:50.200790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198792 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-multus-socket-dir-parent\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.200790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198812 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b65bf7c-bec3-441a-97da-e12835ab7070-device-dir\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.200790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198868 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-run-netns\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.200790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198894 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tljcm\" (UniqueName: \"kubernetes.io/projected/79da16fc-0e3e-44fe-90be-16a0d3d76de3-kube-api-access-tljcm\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.200790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.198939 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eec501bb-6445-4360-8609-7f5a8f557a6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.200790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.199044 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-modprobe-d\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.200790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.199136 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de580cd5-5351-479c-a9ad-0c0936042074-kubelet-config\") pod \"global-pull-secret-syncer-m8lns\" (UID: \"de580cd5-5351-479c-a9ad-0c0936042074\") " pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:50.200790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.199600 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eec501bb-6445-4360-8609-7f5a8f557a6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.201868 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.201826 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/79da16fc-0e3e-44fe-90be-16a0d3d76de3-ovn-node-metrics-cert\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.207633 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.207613 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:14:50.207733 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.207640 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:14:50.207733 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.207653 2573 projected.go:196] Error preparing data for projected volume kube-api-access-gsn77 for pod openshift-network-diagnostics/network-check-target-fh2vt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:50.207733 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.207705 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77 podName:65f1ece3-078e-4c34-9293-e2a4e6007ea8 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:50.707688576 +0000 UTC m=+3.044656751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gsn77" (UniqueName: "kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77") pod "network-check-target-fh2vt" (UID: "65f1ece3-078e-4c34-9293-e2a4e6007ea8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:50.210021 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.209992 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zndxk\" (UniqueName: \"kubernetes.io/projected/ee1912ca-8cb0-4013-bc95-5cb2210bbfe3-kube-api-access-zndxk\") pod \"node-resolver-l7mvd\" (UID: \"ee1912ca-8cb0-4013-bc95-5cb2210bbfe3\") " pod="openshift-dns/node-resolver-l7mvd" Sep 11 14:14:50.210667 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.210642 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmfp9\" (UniqueName: \"kubernetes.io/projected/afa6cfac-b37f-4e64-a308-658d8a3b62d9-kube-api-access-lmfp9\") pod \"network-metrics-daemon-fh5nz\" (UID: \"afa6cfac-b37f-4e64-a308-658d8a3b62d9\") " pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:14:50.210922 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.210893 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-824l6\" (UniqueName: \"kubernetes.io/projected/eec501bb-6445-4360-8609-7f5a8f557a6a-kube-api-access-824l6\") pod \"multus-additional-cni-plugins-jk42w\" (UID: \"eec501bb-6445-4360-8609-7f5a8f557a6a\") " pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.211100 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.211079 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j27cd\" (UniqueName: \"kubernetes.io/projected/9c142fa3-73aa-434a-a3c8-9d3694c03001-kube-api-access-j27cd\") pod \"node-ca-pb9cz\" (UID: \"9c142fa3-73aa-434a-a3c8-9d3694c03001\") " pod="openshift-image-registry/node-ca-pb9cz" Sep 11 14:14:50.212358 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.212332 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwwkv\" (UniqueName: \"kubernetes.io/projected/2b65bf7c-bec3-441a-97da-e12835ab7070-kube-api-access-wwwkv\") pod \"aws-ebs-csi-driver-node-t9q6j\" (UID: \"2b65bf7c-bec3-441a-97da-e12835ab7070\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.213359 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.213153 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tljcm\" (UniqueName: \"kubernetes.io/projected/79da16fc-0e3e-44fe-90be-16a0d3d76de3-kube-api-access-tljcm\") pod \"ovnkube-node-d87x6\" (UID: \"79da16fc-0e3e-44fe-90be-16a0d3d76de3\") " pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.300065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.299994 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-systemd\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.300065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.300040 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-var-lib-cni-bin\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.300065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.300068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-var-lib-cni-multus\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.300301 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.300097 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-var-lib-kubelet\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.300301 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.300121 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-var-lib-cni-bin\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.300301 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.300129 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-systemd\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.300301 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.300126 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d87c49f-6095-47cf-8cc2-736f36cb9a74-konnectivity-ca\") pod \"konnectivity-agent-sn4pw\" (UID: \"7d87c49f-6095-47cf-8cc2-736f36cb9a74\") " pod="kube-system/konnectivity-agent-sn4pw" Sep 11 14:14:50.300301 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.300178 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3612f9b-fe42-47fb-b60e-81d598756204-tmp\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.308790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.308762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-run-multus-certs\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.308916 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.308821 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxh7\" (UniqueName: \"kubernetes.io/projected/80eaab0c-0595-443d-84a6-646467d848dd-kube-api-access-cvxh7\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.308916 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.308880 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-run\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.308916 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.308912 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/545efa3b-e0a6-4c10-bd51-6efd7f0a1a18-host-slash\") pod \"iptables-alerter-gmprw\" (UID: \"545efa3b-e0a6-4c10-bd51-6efd7f0a1a18\") " pod="openshift-network-operator/iptables-alerter-gmprw" Sep 11 14:14:50.309066 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.308933 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-sys\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.309066 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.308951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-cnibin\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.309066 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.308969 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-multus-conf-dir\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.309066 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.308988 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-kubernetes\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.309066 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309004 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-multus-cni-dir\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.309066 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309023 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-run-k8s-cni-cncf-io\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.309066 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309041 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/80eaab0c-0595-443d-84a6-646467d848dd-multus-daemon-config\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.309066 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret\") pod \"global-pull-secret-syncer-m8lns\" (UID: \"de580cd5-5351-479c-a9ad-0c0936042074\") " pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:50.309408 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309083 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-multus-socket-dir-parent\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.309408 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309099 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-run-netns\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.309408 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309104 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d87c49f-6095-47cf-8cc2-736f36cb9a74-konnectivity-ca\") pod \"konnectivity-agent-sn4pw\" (UID: \"7d87c49f-6095-47cf-8cc2-736f36cb9a74\") " pod="kube-system/konnectivity-agent-sn4pw" Sep 11 14:14:50.309408 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-modprobe-d\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.309408 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de580cd5-5351-479c-a9ad-0c0936042074-kubelet-config\") pod \"global-pull-secret-syncer-m8lns\" (UID: \"de580cd5-5351-479c-a9ad-0c0936042074\") " pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:50.309408 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.300190 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-var-lib-kubelet\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.309408 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-sysconfig\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.309408 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309226 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-system-cni-dir\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.309408 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309229 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-kubernetes\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.309408 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309273 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmvr9\" (UniqueName: \"kubernetes.io/projected/545efa3b-e0a6-4c10-bd51-6efd7f0a1a18-kube-api-access-xmvr9\") pod \"iptables-alerter-gmprw\" (UID: \"545efa3b-e0a6-4c10-bd51-6efd7f0a1a18\") " pod="openshift-network-operator/iptables-alerter-gmprw" Sep 11 14:14:50.309408 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309309 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d87c49f-6095-47cf-8cc2-736f36cb9a74-agent-certs\") pod \"konnectivity-agent-sn4pw\" (UID: \"7d87c49f-6095-47cf-8cc2-736f36cb9a74\") " pod="kube-system/konnectivity-agent-sn4pw" Sep 11 14:14:50.309965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309450 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-lib-modules\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.309965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309542 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-var-lib-cni-multus\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.309965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309577 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-system-cni-dir\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.309965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309585 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-lib-modules\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.309965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309622 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3612f9b-fe42-47fb-b60e-81d598756204-etc-tuned\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.309965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309634 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-run-multus-certs\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.309965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309670 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80eaab0c-0595-443d-84a6-646467d848dd-cni-binary-copy\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.309965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-hostroot\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.309965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309920 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-sysctl-conf\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.309965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309949 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-etc-kubernetes\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.310433 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.309981 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de580cd5-5351-479c-a9ad-0c0936042074-dbus\") pod \"global-pull-secret-syncer-m8lns\" (UID: \"de580cd5-5351-479c-a9ad-0c0936042074\") " pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:50.310433 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.310008 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-sysctl-d\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.310433 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.310131 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-sysctl-d\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.310576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.310484 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-sysctl-conf\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.310633 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.310605 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/545efa3b-e0a6-4c10-bd51-6efd7f0a1a18-iptables-alerter-script\") pod \"iptables-alerter-gmprw\" (UID: \"545efa3b-e0a6-4c10-bd51-6efd7f0a1a18\") " pod="openshift-network-operator/iptables-alerter-gmprw" Sep 11 14:14:50.310687 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.310647 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-host\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.310687 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.310672 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80eaab0c-0595-443d-84a6-646467d848dd-cni-binary-copy\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.310687 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.310678 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdpnd\" (UniqueName: \"kubernetes.io/projected/d3612f9b-fe42-47fb-b60e-81d598756204-kube-api-access-zdpnd\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.310854 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.310755 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/545efa3b-e0a6-4c10-bd51-6efd7f0a1a18-host-slash\") pod \"iptables-alerter-gmprw\" (UID: \"545efa3b-e0a6-4c10-bd51-6efd7f0a1a18\") " pod="openshift-network-operator/iptables-alerter-gmprw" Sep 11 14:14:50.310854 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.310821 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-sys\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.310960 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.310887 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-hostroot\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.310960 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.310945 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-host\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.310960 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.310946 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-multus-conf-dir\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.311104 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.310974 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-etc-kubernetes\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.311104 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.310701 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de580cd5-5351-479c-a9ad-0c0936042074-dbus\") pod \"global-pull-secret-syncer-m8lns\" (UID: \"de580cd5-5351-479c-a9ad-0c0936042074\") " pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:50.311104 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.311076 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-var-lib-kubelet\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.311408 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.311383 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-var-lib-kubelet\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.311408 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.311404 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3612f9b-fe42-47fb-b60e-81d598756204-tmp\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.311551 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.311436 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-cnibin\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.311551 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.311504 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-run\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.311551 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.311503 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/545efa3b-e0a6-4c10-bd51-6efd7f0a1a18-iptables-alerter-script\") pod \"iptables-alerter-gmprw\" (UID: \"545efa3b-e0a6-4c10-bd51-6efd7f0a1a18\") " pod="openshift-network-operator/iptables-alerter-gmprw" Sep 11 14:14:50.311703 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.311550 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:50.311703 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.311576 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-run-netns\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.311703 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.311593 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-multus-socket-dir-parent\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.311703 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.311617 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret podName:de580cd5-5351-479c-a9ad-0c0936042074 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:50.811598711 +0000 UTC m=+3.148566881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret") pod "global-pull-secret-syncer-m8lns" (UID: "de580cd5-5351-479c-a9ad-0c0936042074") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:50.311703 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.311671 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-modprobe-d\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.311703 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.311685 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-multus-cni-dir\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.312113 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.312090 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/80eaab0c-0595-443d-84a6-646467d848dd-host-run-k8s-cni-cncf-io\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.312388 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.312369 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/80eaab0c-0595-443d-84a6-646467d848dd-multus-daemon-config\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.312469 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.312410 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de580cd5-5351-479c-a9ad-0c0936042074-kubelet-config\") pod \"global-pull-secret-syncer-m8lns\" (UID: \"de580cd5-5351-479c-a9ad-0c0936042074\") " pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:50.312577 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.312559 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3612f9b-fe42-47fb-b60e-81d598756204-etc-sysconfig\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.312849 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.312818 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d87c49f-6095-47cf-8cc2-736f36cb9a74-agent-certs\") pod \"konnectivity-agent-sn4pw\" (UID: \"7d87c49f-6095-47cf-8cc2-736f36cb9a74\") " pod="kube-system/konnectivity-agent-sn4pw" Sep 11 14:14:50.314191 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.313970 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3612f9b-fe42-47fb-b60e-81d598756204-etc-tuned\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.323307 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.323194 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmvr9\" (UniqueName: \"kubernetes.io/projected/545efa3b-e0a6-4c10-bd51-6efd7f0a1a18-kube-api-access-xmvr9\") pod \"iptables-alerter-gmprw\" (UID: \"545efa3b-e0a6-4c10-bd51-6efd7f0a1a18\") " pod="openshift-network-operator/iptables-alerter-gmprw" Sep 11 14:14:50.323804 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.323758 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxh7\" (UniqueName: \"kubernetes.io/projected/80eaab0c-0595-443d-84a6-646467d848dd-kube-api-access-cvxh7\") pod \"multus-cndlm\" (UID: \"80eaab0c-0595-443d-84a6-646467d848dd\") " pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.325276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.325254 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdpnd\" (UniqueName: \"kubernetes.io/projected/d3612f9b-fe42-47fb-b60e-81d598756204-kube-api-access-zdpnd\") pod \"tuned-4xt8r\" (UID: \"d3612f9b-fe42-47fb-b60e-81d598756204\") " pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.385873 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.385821 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l7mvd" Sep 11 14:14:50.393702 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.393682 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:14:50.397949 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:50.397918 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1912ca_8cb0_4013_bc95_5cb2210bbfe3.slice/crio-5a517f2702bd43faaa436294fadb9ed3addbec7c6e40793992a19b4dc6876b0b WatchSource:0}: Error finding container 5a517f2702bd43faaa436294fadb9ed3addbec7c6e40793992a19b4dc6876b0b: Status 404 returned error can't find the container with id 5a517f2702bd43faaa436294fadb9ed3addbec7c6e40793992a19b4dc6876b0b Sep 11 14:14:50.401301 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.401281 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pb9cz" Sep 11 14:14:50.403376 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:50.403352 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79da16fc_0e3e_44fe_90be_16a0d3d76de3.slice/crio-68061ecbe2041b83e9acc713f46ba69445585ba1783a091e796c5f1b5386337b WatchSource:0}: Error finding container 68061ecbe2041b83e9acc713f46ba69445585ba1783a091e796c5f1b5386337b: Status 404 returned error can't find the container with id 68061ecbe2041b83e9acc713f46ba69445585ba1783a091e796c5f1b5386337b Sep 11 14:14:50.407437 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.407417 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jk42w" Sep 11 14:14:50.412113 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:50.412092 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c142fa3_73aa_434a_a3c8_9d3694c03001.slice/crio-c0e0e6eee29d599a68eb7e8ede14d1fb0706ace947af265d68a96b84ac90ca9f WatchSource:0}: Error finding container c0e0e6eee29d599a68eb7e8ede14d1fb0706ace947af265d68a96b84ac90ca9f: Status 404 returned error can't find the container with id c0e0e6eee29d599a68eb7e8ede14d1fb0706ace947af265d68a96b84ac90ca9f Sep 11 14:14:50.415090 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.415033 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" Sep 11 14:14:50.416735 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:50.416694 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec501bb_6445_4360_8609_7f5a8f557a6a.slice/crio-ecca89aa5e4a18c72d65abfc9586f91d969a35da17f24c145a6939f64766e0cb WatchSource:0}: Error finding container ecca89aa5e4a18c72d65abfc9586f91d969a35da17f24c145a6939f64766e0cb: Status 404 returned error can't find the container with id ecca89aa5e4a18c72d65abfc9586f91d969a35da17f24c145a6939f64766e0cb Sep 11 14:14:50.422198 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.422175 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gmprw" Sep 11 14:14:50.428134 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:50.428109 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b65bf7c_bec3_441a_97da_e12835ab7070.slice/crio-fd850500e9ef4d7d234ebfe7672cf163d8b6725aa890d99439e4160177e9e9f1 WatchSource:0}: Error finding container fd850500e9ef4d7d234ebfe7672cf163d8b6725aa890d99439e4160177e9e9f1: Status 404 returned error can't find the container with id fd850500e9ef4d7d234ebfe7672cf163d8b6725aa890d99439e4160177e9e9f1 Sep 11 14:14:50.431909 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.431748 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sn4pw" Sep 11 14:14:50.434999 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:50.434977 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545efa3b_e0a6_4c10_bd51_6efd7f0a1a18.slice/crio-0438216d3efd9441c511c9e6d59ae494f7b2a476a3601082e7d154197d1bede0 WatchSource:0}: Error finding container 0438216d3efd9441c511c9e6d59ae494f7b2a476a3601082e7d154197d1bede0: Status 404 returned error can't find the container with id 0438216d3efd9441c511c9e6d59ae494f7b2a476a3601082e7d154197d1bede0 Sep 11 14:14:50.437768 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.437746 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" Sep 11 14:14:50.443034 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.443012 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cndlm" Sep 11 14:14:50.454722 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:14:50.454702 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80eaab0c_0595_443d_84a6_646467d848dd.slice/crio-912e3c2570e7c6981e06ff3c80db638742fece608ee348669fb3df3032ea80f9 WatchSource:0}: Error finding container 912e3c2570e7c6981e06ff3c80db638742fece608ee348669fb3df3032ea80f9: Status 404 returned error can't find the container with id 912e3c2570e7c6981e06ff3c80db638742fece608ee348669fb3df3032ea80f9 Sep 11 14:14:50.713572 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.713486 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs\") pod \"network-metrics-daemon-fh5nz\" (UID: \"afa6cfac-b37f-4e64-a308-658d8a3b62d9\") " pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:14:50.713572 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.713528 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsn77\" (UniqueName: \"kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77\") pod \"network-check-target-fh2vt\" (UID: \"65f1ece3-078e-4c34-9293-e2a4e6007ea8\") " pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:14:50.713783 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.713643 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:50.713783 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.713657 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:14:50.713783 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.713674 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:14:50.713783 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.713686 2573 projected.go:196] Error preparing data for projected volume kube-api-access-gsn77 for pod openshift-network-diagnostics/network-check-target-fh2vt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:50.713783 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.713708 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs podName:afa6cfac-b37f-4e64-a308-658d8a3b62d9 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:51.713689316 +0000 UTC m=+4.050657508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs") pod "network-metrics-daemon-fh5nz" (UID: "afa6cfac-b37f-4e64-a308-658d8a3b62d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:50.713783 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.713726 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77 podName:65f1ece3-078e-4c34-9293-e2a4e6007ea8 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:51.713716556 +0000 UTC m=+4.050684725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gsn77" (UniqueName: "kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77") pod "network-check-target-fh2vt" (UID: "65f1ece3-078e-4c34-9293-e2a4e6007ea8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:50.813994 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:50.813952 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret\") pod \"global-pull-secret-syncer-m8lns\" (UID: \"de580cd5-5351-479c-a9ad-0c0936042074\") " pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:50.814172 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.814147 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:50.814239 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:50.814228 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret podName:de580cd5-5351-479c-a9ad-0c0936042074 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:51.81420619 +0000 UTC m=+4.151174362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret") pod "global-pull-secret-syncer-m8lns" (UID: "de580cd5-5351-479c-a9ad-0c0936042074") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:51.136448 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:51.136361 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 14:09:50 +0000 UTC" deadline="2028-02-27 11:09:24.690180514 +0000 UTC" Sep 11 14:14:51.136448 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:51.136401 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12812h54m33.553786404s" Sep 11 14:14:51.191925 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:51.191871 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l7mvd" event={"ID":"ee1912ca-8cb0-4013-bc95-5cb2210bbfe3","Type":"ContainerStarted","Data":"5a517f2702bd43faaa436294fadb9ed3addbec7c6e40793992a19b4dc6876b0b"} Sep 11 14:14:51.193222 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:51.193193 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cndlm" event={"ID":"80eaab0c-0595-443d-84a6-646467d848dd","Type":"ContainerStarted","Data":"912e3c2570e7c6981e06ff3c80db638742fece608ee348669fb3df3032ea80f9"} Sep 11 14:14:51.194469 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:51.194443 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gmprw" event={"ID":"545efa3b-e0a6-4c10-bd51-6efd7f0a1a18","Type":"ContainerStarted","Data":"0438216d3efd9441c511c9e6d59ae494f7b2a476a3601082e7d154197d1bede0"} Sep 11 14:14:51.195810 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:51.195785 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" event={"ID":"2b65bf7c-bec3-441a-97da-e12835ab7070","Type":"ContainerStarted","Data":"fd850500e9ef4d7d234ebfe7672cf163d8b6725aa890d99439e4160177e9e9f1"} Sep 11 14:14:51.197207 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:51.197177 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk42w" event={"ID":"eec501bb-6445-4360-8609-7f5a8f557a6a","Type":"ContainerStarted","Data":"ecca89aa5e4a18c72d65abfc9586f91d969a35da17f24c145a6939f64766e0cb"} Sep 11 14:14:51.198545 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:51.198521 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" event={"ID":"79da16fc-0e3e-44fe-90be-16a0d3d76de3","Type":"ContainerStarted","Data":"68061ecbe2041b83e9acc713f46ba69445585ba1783a091e796c5f1b5386337b"} Sep 11 14:14:51.199869 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:51.199845 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" event={"ID":"d3612f9b-fe42-47fb-b60e-81d598756204","Type":"ContainerStarted","Data":"679d6c6f89ffc87830ab0b3fb08c35264fa1f4b7a7bbc34902571f6fe66a2fee"} Sep 11 14:14:51.201128 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:51.201098 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sn4pw" event={"ID":"7d87c49f-6095-47cf-8cc2-736f36cb9a74","Type":"ContainerStarted","Data":"1f8e92732ad62ec794a6f743dbe78f846a8ea0e2452fea5f4caac482d495bc4b"} Sep 11 14:14:51.202301 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:51.202268 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pb9cz" event={"ID":"9c142fa3-73aa-434a-a3c8-9d3694c03001","Type":"ContainerStarted","Data":"c0e0e6eee29d599a68eb7e8ede14d1fb0706ace947af265d68a96b84ac90ca9f"} Sep 11 14:14:51.721774 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:51.721738 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs\") pod \"network-metrics-daemon-fh5nz\" (UID: \"afa6cfac-b37f-4e64-a308-658d8a3b62d9\") " pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:14:51.721774 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:51.721798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsn77\" (UniqueName: \"kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77\") pod \"network-check-target-fh2vt\" (UID: \"65f1ece3-078e-4c34-9293-e2a4e6007ea8\") " pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:14:51.722158 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:51.722006 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:14:51.722158 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:51.722028 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:14:51.722158 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:51.722041 2573 projected.go:196] Error preparing data for projected volume kube-api-access-gsn77 for pod openshift-network-diagnostics/network-check-target-fh2vt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:51.722158 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:51.722100 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77 podName:65f1ece3-078e-4c34-9293-e2a4e6007ea8 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:53.722081149 +0000 UTC m=+6.059049338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gsn77" (UniqueName: "kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77") pod "network-check-target-fh2vt" (UID: "65f1ece3-078e-4c34-9293-e2a4e6007ea8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:51.722509 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:51.722488 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:51.722569 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:51.722538 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs podName:afa6cfac-b37f-4e64-a308-658d8a3b62d9 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:53.722522762 +0000 UTC m=+6.059490935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs") pod "network-metrics-daemon-fh5nz" (UID: "afa6cfac-b37f-4e64-a308-658d8a3b62d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:51.822715 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:51.822669 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret\") pod \"global-pull-secret-syncer-m8lns\" (UID: \"de580cd5-5351-479c-a9ad-0c0936042074\") " pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:51.822916 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:51.822899 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:51.822987 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:51.822967 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret podName:de580cd5-5351-479c-a9ad-0c0936042074 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:53.822949757 +0000 UTC m=+6.159917930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret") pod "global-pull-secret-syncer-m8lns" (UID: "de580cd5-5351-479c-a9ad-0c0936042074") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:52.137572 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:52.137488 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 14:09:50 +0000 UTC" deadline="2028-05-30 10:58:16.891857952 +0000 UTC" Sep 11 14:14:52.137572 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:52.137529 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15044h43m24.754333036s" Sep 11 14:14:52.183586 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:52.183563 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:52.183699 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:52.183677 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:14:52.183774 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:52.183756 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:14:52.183872 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:52.183853 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:14:52.183929 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:52.183913 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:14:52.184044 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:52.184026 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:14:53.219017 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:53.218964 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal" event={"ID":"344072aebe26cd1a15ff23a2d0a3fde8","Type":"ContainerStarted","Data":"fe13b538d2dabb5863c9d352e65b9e64b7729f4542387ad0d6fe0f7954bce860"} Sep 11 14:14:53.233924 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:53.233840 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.ec2.internal" event={"ID":"ecb7720b7510a32693de89b52bd85266","Type":"ContainerStarted","Data":"03a70c0661ad8c0ec61a108956671e1363c619fb58018aa382b15864a3578719"} Sep 11 14:14:53.739767 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:53.739732 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs\") pod \"network-metrics-daemon-fh5nz\" (UID: \"afa6cfac-b37f-4e64-a308-658d8a3b62d9\") " pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:14:53.739953 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:53.739795 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsn77\" (UniqueName: \"kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77\") pod \"network-check-target-fh2vt\" (UID: \"65f1ece3-078e-4c34-9293-e2a4e6007ea8\") " pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:14:53.740019 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:53.739965 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:14:53.740019 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:53.739986 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:14:53.740019 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:53.739999 2573 projected.go:196] Error preparing data for projected volume kube-api-access-gsn77 for pod openshift-network-diagnostics/network-check-target-fh2vt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:53.740174 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:53.740055 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77 podName:65f1ece3-078e-4c34-9293-e2a4e6007ea8 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:57.740037466 +0000 UTC m=+10.077005641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gsn77" (UniqueName: "kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77") pod "network-check-target-fh2vt" (UID: "65f1ece3-078e-4c34-9293-e2a4e6007ea8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:53.740487 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:53.740465 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:53.740573 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:53.740527 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs podName:afa6cfac-b37f-4e64-a308-658d8a3b62d9 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:57.740511144 +0000 UTC m=+10.077479318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs") pod "network-metrics-daemon-fh5nz" (UID: "afa6cfac-b37f-4e64-a308-658d8a3b62d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:53.841153 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:53.841091 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret\") pod \"global-pull-secret-syncer-m8lns\" (UID: \"de580cd5-5351-479c-a9ad-0c0936042074\") " pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:53.841321 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:53.841271 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:53.841391 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:53.841334 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret podName:de580cd5-5351-479c-a9ad-0c0936042074 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:57.841315432 +0000 UTC m=+10.178283619 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret") pod "global-pull-secret-syncer-m8lns" (UID: "de580cd5-5351-479c-a9ad-0c0936042074") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:54.183993 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:54.183173 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:54.183993 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:54.183313 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:14:54.183993 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:54.183395 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:14:54.183993 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:54.183497 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:14:54.183993 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:54.183541 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:14:54.183993 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:54.183593 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:14:56.181970 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:56.181574 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:56.181970 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:56.181574 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:14:56.181970 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:56.181710 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:14:56.181970 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:56.181728 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:14:56.181970 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:56.181848 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:14:56.181970 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:56.181928 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:14:57.246845 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:57.246786 2573 generic.go:356] "Generic (PLEG): container finished" podID="344072aebe26cd1a15ff23a2d0a3fde8" containerID="fe13b538d2dabb5863c9d352e65b9e64b7729f4542387ad0d6fe0f7954bce860" exitCode=0 Sep 11 14:14:57.247299 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:57.246859 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal" event={"ID":"344072aebe26cd1a15ff23a2d0a3fde8","Type":"ContainerDied","Data":"fe13b538d2dabb5863c9d352e65b9e64b7729f4542387ad0d6fe0f7954bce860"} Sep 11 14:14:57.267212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:57.267150 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-110.ec2.internal" podStartSLOduration=8.266699883 podStartE2EDuration="8.266699883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:14:53.259731566 +0000 UTC m=+5.596699770" watchObservedRunningTime="2026-09-11 14:14:57.266699883 +0000 UTC m=+9.603668075" Sep 11 14:14:57.776717 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:57.775963 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsn77\" (UniqueName: \"kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77\") pod \"network-check-target-fh2vt\" (UID: \"65f1ece3-078e-4c34-9293-e2a4e6007ea8\") " pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:14:57.776717 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:57.776070 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs\") pod \"network-metrics-daemon-fh5nz\" (UID: \"afa6cfac-b37f-4e64-a308-658d8a3b62d9\") " pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:14:57.776717 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:57.776190 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:57.776717 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:57.776243 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs podName:afa6cfac-b37f-4e64-a308-658d8a3b62d9 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:05.776225453 +0000 UTC m=+18.113193622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs") pod "network-metrics-daemon-fh5nz" (UID: "afa6cfac-b37f-4e64-a308-658d8a3b62d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:57.776717 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:57.776324 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:14:57.776717 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:57.776339 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:14:57.776717 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:57.776351 2573 projected.go:196] Error preparing data for projected volume kube-api-access-gsn77 for pod openshift-network-diagnostics/network-check-target-fh2vt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:57.776717 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:57.776390 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77 podName:65f1ece3-078e-4c34-9293-e2a4e6007ea8 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:05.776375971 +0000 UTC m=+18.113344146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gsn77" (UniqueName: "kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77") pod "network-check-target-fh2vt" (UID: "65f1ece3-078e-4c34-9293-e2a4e6007ea8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:57.876434 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:57.876395 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret\") pod \"global-pull-secret-syncer-m8lns\" (UID: \"de580cd5-5351-479c-a9ad-0c0936042074\") " pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:57.876618 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:57.876594 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:57.876767 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:57.876669 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret podName:de580cd5-5351-479c-a9ad-0c0936042074 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:05.876651393 +0000 UTC m=+18.213619566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret") pod "global-pull-secret-syncer-m8lns" (UID: "de580cd5-5351-479c-a9ad-0c0936042074") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:58.185116 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:58.185030 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:14:58.185116 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:58.185045 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:14:58.185312 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:14:58.185143 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:14:58.185312 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:58.185150 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:14:58.185312 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:58.185233 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:14:58.185439 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:14:58.185322 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:15:00.184924 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:00.184856 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:15:00.184924 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:00.184886 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:00.185419 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:00.184978 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:00.185419 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:00.184981 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:15:00.185419 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:00.185087 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:15:00.185419 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:00.185188 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:15:02.184371 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:02.184338 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:02.184371 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:02.184374 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:02.184904 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:02.184335 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:15:02.184904 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:02.184439 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:15:02.184904 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:02.184545 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:15:02.184904 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:02.184608 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:15:04.181346 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:04.181303 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:04.181814 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:04.181310 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:04.181814 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:04.181427 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:15:04.181814 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:04.181318 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:15:04.181814 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:04.181503 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:15:04.181814 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:04.181592 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:15:05.829355 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:05.829318 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs\") pod \"network-metrics-daemon-fh5nz\" (UID: \"afa6cfac-b37f-4e64-a308-658d8a3b62d9\") " pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:15:05.829863 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:05.829370 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsn77\" (UniqueName: \"kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77\") pod \"network-check-target-fh2vt\" (UID: \"65f1ece3-078e-4c34-9293-e2a4e6007ea8\") " pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:05.829863 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:05.829498 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:05.829863 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:05.829544 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:05.829863 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:05.829565 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs podName:afa6cfac-b37f-4e64-a308-658d8a3b62d9 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:21.829548077 +0000 UTC m=+34.166516253 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs") pod "network-metrics-daemon-fh5nz" (UID: "afa6cfac-b37f-4e64-a308-658d8a3b62d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:05.829863 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:05.829569 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:05.829863 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:05.829583 2573 projected.go:196] Error preparing data for projected volume kube-api-access-gsn77 for pod openshift-network-diagnostics/network-check-target-fh2vt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:05.829863 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:05.829636 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77 podName:65f1ece3-078e-4c34-9293-e2a4e6007ea8 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:21.829619892 +0000 UTC m=+34.166588077 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gsn77" (UniqueName: "kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77") pod "network-check-target-fh2vt" (UID: "65f1ece3-078e-4c34-9293-e2a4e6007ea8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:05.930250 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:05.930213 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret\") pod \"global-pull-secret-syncer-m8lns\" (UID: \"de580cd5-5351-479c-a9ad-0c0936042074\") " pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:05.930399 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:05.930367 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:05.930477 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:05.930440 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret podName:de580cd5-5351-479c-a9ad-0c0936042074 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:21.930420984 +0000 UTC m=+34.267389176 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret") pod "global-pull-secret-syncer-m8lns" (UID: "de580cd5-5351-479c-a9ad-0c0936042074") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:06.181624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:06.181091 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:15:06.181624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:06.181098 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:06.181624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:06.181234 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:06.181624 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:06.181236 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:15:06.181624 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:06.181333 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:15:06.181624 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:06.181413 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:15:08.181219 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:08.181180 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:08.181664 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:08.181316 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:08.181664 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:08.181335 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:15:08.181664 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:08.181405 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:15:08.181664 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:08.181469 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:15:08.181664 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:08.181546 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:15:09.269055 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.268902 2573 generic.go:356] "Generic (PLEG): container finished" podID="eec501bb-6445-4360-8609-7f5a8f557a6a" containerID="f4799ec68ca196d9fd2b03341053a21ca650998f077f6a93a1f3cbdfd2ea0184" exitCode=0 Sep 11 14:15:09.269765 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.268935 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk42w" event={"ID":"eec501bb-6445-4360-8609-7f5a8f557a6a","Type":"ContainerDied","Data":"f4799ec68ca196d9fd2b03341053a21ca650998f077f6a93a1f3cbdfd2ea0184"} Sep 11 14:15:09.271457 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.271432 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" event={"ID":"79da16fc-0e3e-44fe-90be-16a0d3d76de3","Type":"ContainerStarted","Data":"f30230cbfa4c16030b47a0343c2bc446ff148ba11b77ebd72c4be348f9f027ca"} Sep 11 14:15:09.271560 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.271515 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" event={"ID":"79da16fc-0e3e-44fe-90be-16a0d3d76de3","Type":"ContainerStarted","Data":"4ff463a8e47b60024ddd49fb1569658b9ec3e6c9d4a01fe654d9ee86adf701fa"} Sep 11 14:15:09.273420 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.273396 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal" event={"ID":"344072aebe26cd1a15ff23a2d0a3fde8","Type":"ContainerStarted","Data":"6d0c2a7324a3e815419f47b5eac7d59c25c1b30fa4707f301df7b114ed04e815"} Sep 11 14:15:09.275279 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.275257 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" event={"ID":"d3612f9b-fe42-47fb-b60e-81d598756204","Type":"ContainerStarted","Data":"6c1573233d7ae15785f26a35a03704e54faf77287a4303c1f96f299b22827283"} Sep 11 14:15:09.276689 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.276662 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sn4pw" event={"ID":"7d87c49f-6095-47cf-8cc2-736f36cb9a74","Type":"ContainerStarted","Data":"ca1514d0bac74f1121aa86096e5cd28d888949961d31e27172eb4da5d9ecbc80"} Sep 11 14:15:09.278312 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.278293 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pb9cz" event={"ID":"9c142fa3-73aa-434a-a3c8-9d3694c03001","Type":"ContainerStarted","Data":"f81c0404426e2e4ba3f1bd90fca35c064c4c785ae0a21be6235cbf084bfbea8e"} Sep 11 14:15:09.279584 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.279567 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l7mvd" event={"ID":"ee1912ca-8cb0-4013-bc95-5cb2210bbfe3","Type":"ContainerStarted","Data":"754f9f1ec50eae4e615eebed699e89994764962c91b6dc99426d1dd5e18935d7"} Sep 11 14:15:09.280940 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.280916 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cndlm" event={"ID":"80eaab0c-0595-443d-84a6-646467d848dd","Type":"ContainerStarted","Data":"67f7148a4e10e3246447f52b4789e3cd9f2b26b3c528316d3fe03304761a897b"} Sep 11 14:15:09.282425 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.282404 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" event={"ID":"2b65bf7c-bec3-441a-97da-e12835ab7070","Type":"ContainerStarted","Data":"d8a26c3133d92f6b1f29f3fbfc615dd51bd49fca2d47a62099794df4b21e3bfa"} Sep 11 14:15:09.313441 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.313399 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sn4pw" podStartSLOduration=10.460763579 podStartE2EDuration="21.313387501s" totalImagesPullingTime="10.852623922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:09.31299636 +0000 UTC m=+21.649964546" watchObservedRunningTime="2026-09-11 14:15:09.313387501 +0000 UTC m=+21.650355689" Sep 11 14:15:09.352847 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.352804 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4xt8r" podStartSLOduration=3.137252818 podStartE2EDuration="21.352791218s" totalImagesPullingTime="18.2155384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:09.352734639 +0000 UTC m=+21.689702831" watchObservedRunningTime="2026-09-11 14:15:09.352791218 +0000 UTC m=+21.689759387" Sep 11 14:15:09.353198 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.353176 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pb9cz" podStartSLOduration=3.214367536 podStartE2EDuration="21.353170684s" totalImagesPullingTime="18.138803148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:09.329134918 +0000 UTC m=+21.666103109" watchObservedRunningTime="2026-09-11 14:15:09.353170684 +0000 UTC m=+21.690138875" Sep 11 14:15:09.402783 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.402764 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sn4pw" Sep 11 14:15:09.403321 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.403305 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sn4pw" Sep 11 14:15:09.422847 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.422791 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cndlm" podStartSLOduration=3.214357518 podStartE2EDuration="21.422771975s" totalImagesPullingTime="18.208414457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:09.3808252 +0000 UTC m=+21.717793391" watchObservedRunningTime="2026-09-11 14:15:09.422771975 +0000 UTC m=+21.759740165" Sep 11 14:15:09.423381 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.423345 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-110.ec2.internal" podStartSLOduration=20.423336093 podStartE2EDuration="20.423336093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:09.423107329 +0000 UTC m=+21.760075521" watchObservedRunningTime="2026-09-11 14:15:09.423336093 +0000 UTC m=+21.760304286" Sep 11 14:15:09.471088 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:09.471049 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l7mvd" podStartSLOduration=3.308786518 podStartE2EDuration="21.47102936s" totalImagesPullingTime="18.162242842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:09.445169028 +0000 UTC m=+21.782137232" watchObservedRunningTime="2026-09-11 14:15:09.47102936 +0000 UTC m=+21.807997552" Sep 11 14:15:10.181416 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:10.181385 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:10.181565 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:10.181386 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:10.181565 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:10.181479 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:15:10.181565 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:10.181508 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:15:10.181668 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:10.181597 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:15:10.181709 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:10.181692 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:15:10.285680 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:10.285652 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gmprw" event={"ID":"545efa3b-e0a6-4c10-bd51-6efd7f0a1a18","Type":"ContainerStarted","Data":"46e8287a43bba64d38985ea86280aaf8f0af83b8b37594e1c038c638be23cbd2"} Sep 11 14:15:10.288639 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:10.288615 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" event={"ID":"79da16fc-0e3e-44fe-90be-16a0d3d76de3","Type":"ContainerStarted","Data":"f97f48705b0aded7cff9f060cc79ad9cceea5ab434334d2536e3f5482c8c93c3"} Sep 11 14:15:10.288753 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:10.288673 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" event={"ID":"79da16fc-0e3e-44fe-90be-16a0d3d76de3","Type":"ContainerStarted","Data":"87474d3cdd30ec3039024ed9398d77ad6a553cec616b9bba9b48fda3cda42b62"} Sep 11 14:15:10.288753 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:10.288702 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" event={"ID":"79da16fc-0e3e-44fe-90be-16a0d3d76de3","Type":"ContainerStarted","Data":"ee42aa0515b65a8abe223990b300c9b095e9a7b143a313481e95f3eb5419a79c"} Sep 11 14:15:10.288753 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:10.288729 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" event={"ID":"79da16fc-0e3e-44fe-90be-16a0d3d76de3","Type":"ContainerStarted","Data":"8af0973aea554e9272fe065b5787171267032f774da8651b38df77e275e9a01f"} Sep 11 14:15:10.301325 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:10.301287 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gmprw" podStartSLOduration=4.074504543 podStartE2EDuration="22.301273697s" totalImagesPullingTime="18.226769154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:10.300960039 +0000 UTC m=+22.637928230" watchObservedRunningTime="2026-09-11 14:15:10.301273697 +0000 UTC m=+22.638241888" Sep 11 14:15:10.422441 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:10.422421 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 14:15:11.140994 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:11.140823 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T14:15:10.422436212Z","UUID":"3701b137-b0b3-4b78-8e93-cc91bc7a65c7","Handler":null,"Name":"","Endpoint":""} Sep 11 14:15:11.142632 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:11.142608 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 14:15:11.142762 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:11.142647 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 14:15:11.293594 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:11.293552 2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 14:15:11.293594 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:11.293567 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" event={"ID":"2b65bf7c-bec3-441a-97da-e12835ab7070","Type":"ContainerStarted","Data":"56bbb7cb48db74a0a83e4257a177c8e8f572da729bbdc08e111cd9f49bef6226"} Sep 11 14:15:12.180639 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:12.180610 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:12.180934 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:12.180618 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:15:12.180934 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:12.180733 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:15:12.180934 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:12.180618 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:12.180934 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:12.180872 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:15:12.181125 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:12.180946 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:15:12.297668 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:12.297635 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" event={"ID":"2b65bf7c-bec3-441a-97da-e12835ab7070","Type":"ContainerStarted","Data":"efa43c064c3f154c1bdce3fd97fc4b3f7339b0a65f5286a1cd8cb4480509e76e"} Sep 11 14:15:12.300853 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:12.300817 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" event={"ID":"79da16fc-0e3e-44fe-90be-16a0d3d76de3","Type":"ContainerStarted","Data":"8ddca7cb68638fd26e41d11bf1108b87773ff501bed89faa7df5a547033ce36a"} Sep 11 14:15:12.318188 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:12.318147 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9q6j" podStartSLOduration=3.7142533650000003 podStartE2EDuration="24.318134921s" totalImagesPullingTime="20.603881556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:12.317680332 +0000 UTC m=+24.654648535" watchObservedRunningTime="2026-09-11 14:15:12.318134921 +0000 UTC m=+24.655103111" Sep 11 14:15:14.181361 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:14.181131 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:15:14.181815 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:14.181130 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:14.181815 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:14.181474 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:15:14.181815 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:14.181134 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:14.181815 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:14.181553 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:15:14.181815 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:14.181631 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:15:15.311539 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:15.311513 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" event={"ID":"79da16fc-0e3e-44fe-90be-16a0d3d76de3","Type":"ContainerStarted","Data":"2a61e803a208cba2c8aa6c61d1774e2337e73297b4ba06e671a0ea6d87fcf9d7"} Sep 11 14:15:15.312272 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:15.311770 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:15:15.326592 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:15.326571 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:15:15.437244 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:15.437205 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" podStartSLOduration=8.811987767 podStartE2EDuration="27.437190191s" totalImagesPullingTime="18.625202424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:15.367906132 +0000 UTC m=+27.704874317" watchObservedRunningTime="2026-09-11 14:15:15.437190191 +0000 UTC m=+27.774158381" Sep 11 14:15:16.181387 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:16.181355 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:16.181580 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:16.181355 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:16.181580 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:16.181451 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:15:16.181580 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:16.181470 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:15:16.181580 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:16.181554 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:15:16.181760 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:16.181652 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:15:16.315686 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:16.315649 2573 generic.go:356] "Generic (PLEG): container finished" podID="eec501bb-6445-4360-8609-7f5a8f557a6a" containerID="8954c8e67fa59a6b2e8eda0a71c5e6c6795c83ccef94b6acf65a5584c178e070" exitCode=0 Sep 11 14:15:16.317247 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:16.317218 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk42w" event={"ID":"eec501bb-6445-4360-8609-7f5a8f557a6a","Type":"ContainerDied","Data":"8954c8e67fa59a6b2e8eda0a71c5e6c6795c83ccef94b6acf65a5584c178e070"} Sep 11 14:15:16.317367 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:16.317275 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:15:16.317414 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:16.317399 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:15:16.337941 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:16.337920 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:15:16.976033 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:16.975777 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m8lns"] Sep 11 14:15:16.976195 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:16.976069 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:16.976195 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:16.976151 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:15:16.983853 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:16.983811 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fh2vt"] Sep 11 14:15:16.984009 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:16.983923 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:16.984065 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:16.984006 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:15:16.984296 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:16.984272 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fh5nz"] Sep 11 14:15:16.984395 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:16.984379 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:15:16.984511 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:16.984486 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:15:18.181935 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:18.181853 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:18.182243 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:18.181934 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:15:18.323098 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:18.323066 2573 generic.go:356] "Generic (PLEG): container finished" podID="eec501bb-6445-4360-8609-7f5a8f557a6a" containerID="cd1e51bb66a7860b82e6173f8e5e06e8317c0557aca19ceb3b5c42bb7f456e3d" exitCode=0 Sep 11 14:15:18.323258 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:18.323147 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk42w" event={"ID":"eec501bb-6445-4360-8609-7f5a8f557a6a","Type":"ContainerDied","Data":"cd1e51bb66a7860b82e6173f8e5e06e8317c0557aca19ceb3b5c42bb7f456e3d"} Sep 11 14:15:19.180991 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:19.180964 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:19.181146 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:19.181074 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:15:19.181146 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:19.181125 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:15:19.181252 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:19.181210 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:15:20.180867 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:20.180788 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:20.181226 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:20.180895 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m8lns" podUID="de580cd5-5351-479c-a9ad-0c0936042074" Sep 11 14:15:20.246558 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:20.246524 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sn4pw" Sep 11 14:15:20.246701 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:20.246628 2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 14:15:20.247027 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:20.247014 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sn4pw" Sep 11 14:15:20.329815 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:20.329786 2573 generic.go:356] "Generic (PLEG): container finished" podID="eec501bb-6445-4360-8609-7f5a8f557a6a" containerID="07e40d77ac37a7fc133a91e46fbf3d8e3149a483193e07c6e7b84659a1751ad4" exitCode=0 Sep 11 14:15:20.329955 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:20.329860 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk42w" event={"ID":"eec501bb-6445-4360-8609-7f5a8f557a6a","Type":"ContainerDied","Data":"07e40d77ac37a7fc133a91e46fbf3d8e3149a483193e07c6e7b84659a1751ad4"} Sep 11 14:15:21.181348 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.181151 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:21.181811 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.181209 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:15:21.181811 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:21.181435 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fh2vt" podUID="65f1ece3-078e-4c34-9293-e2a4e6007ea8" Sep 11 14:15:21.181811 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:21.181516 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh5nz" podUID="afa6cfac-b37f-4e64-a308-658d8a3b62d9" Sep 11 14:15:21.489049 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.488971 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-110.ec2.internal" event="NodeReady" Sep 11 14:15:21.489197 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.489124 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 14:15:21.538599 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.538566 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-856c4b595b-4k97s"] Sep 11 14:15:21.540494 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.540473 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.544596 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.544568 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 14:15:21.544596 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.544580 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 14:15:21.544903 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.544888 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jb4sz\"" Sep 11 14:15:21.544982 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.544942 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 14:15:21.550629 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.550610 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 14:15:21.556533 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.556516 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-856c4b595b-4k97s"] Sep 11 14:15:21.561297 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.561273 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-g5n7k"] Sep 11 14:15:21.563084 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.563058 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4gpp4"] Sep 11 14:15:21.563187 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.563170 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g5n7k" Sep 11 14:15:21.564947 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.564929 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:21.566133 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.566113 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 14:15:21.569559 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.566666 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 14:15:21.569559 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.567031 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 14:15:21.569559 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.567251 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bjq4z\"" Sep 11 14:15:21.569559 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.567521 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 14:15:21.569559 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.567759 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 14:15:21.569559 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.568016 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xblvp\"" Sep 11 14:15:21.577477 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.577451 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g5n7k"] Sep 11 14:15:21.582136 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.582114 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4gpp4"] Sep 11 14:15:21.646511 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.646480 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:21.646511 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.646514 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.646721 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.646537 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csmpw\" (UniqueName: \"kubernetes.io/projected/4f4f4176-30cd-40b6-bc5d-3f1c1d2cd3ff-kube-api-access-csmpw\") pod \"ingress-canary-g5n7k\" (UID: \"4f4f4176-30cd-40b6-bc5d-3f1c1d2cd3ff\") " pod="openshift-ingress-canary/ingress-canary-g5n7k" Sep 11 14:15:21.646721 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.646649 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32273522-7dd5-42c8-b2ce-fb3c46906000-trusted-ca\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.646721 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.646686 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-bound-sa-token\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.646721 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.646713 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68438637-9ab0-4bfe-9074-767a01a364b5-config-volume\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:21.646930 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.646741 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqk57\" (UniqueName: \"kubernetes.io/projected/68438637-9ab0-4bfe-9074-767a01a364b5-kube-api-access-cqk57\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:21.646930 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.646781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68438637-9ab0-4bfe-9074-767a01a364b5-tmp-dir\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:21.646930 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.646806 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32273522-7dd5-42c8-b2ce-fb3c46906000-ca-trust-extracted\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.646930 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.646847 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wknfs\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-kube-api-access-wknfs\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.646930 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.646882 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f4f4176-30cd-40b6-bc5d-3f1c1d2cd3ff-cert\") pod \"ingress-canary-g5n7k\" (UID: \"4f4f4176-30cd-40b6-bc5d-3f1c1d2cd3ff\") " pod="openshift-ingress-canary/ingress-canary-g5n7k" Sep 11 14:15:21.647150 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.646944 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-certificates\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.647150 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.646978 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32273522-7dd5-42c8-b2ce-fb3c46906000-installation-pull-secrets\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.647150 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.647005 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32273522-7dd5-42c8-b2ce-fb3c46906000-image-registry-private-configuration\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.747878 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.747844 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:21.748010 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.747887 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.748010 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.747916 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csmpw\" (UniqueName: \"kubernetes.io/projected/4f4f4176-30cd-40b6-bc5d-3f1c1d2cd3ff-kube-api-access-csmpw\") pod \"ingress-canary-g5n7k\" (UID: \"4f4f4176-30cd-40b6-bc5d-3f1c1d2cd3ff\") " pod="openshift-ingress-canary/ingress-canary-g5n7k" Sep 11 14:15:21.748010 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:21.747998 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:21.748187 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:21.748017 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:15:21.748187 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:21.748037 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-856c4b595b-4k97s: secret "image-registry-tls" not found Sep 11 14:15:21.748187 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:21.748071 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls podName:68438637-9ab0-4bfe-9074-767a01a364b5 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:22.248048601 +0000 UTC m=+34.585016785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls") pod "dns-default-4gpp4" (UID: "68438637-9ab0-4bfe-9074-767a01a364b5") : secret "dns-default-metrics-tls" not found Sep 11 14:15:21.748187 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:21.748100 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls podName:32273522-7dd5-42c8-b2ce-fb3c46906000 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:22.248082769 +0000 UTC m=+34.585050943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls") pod "image-registry-856c4b595b-4k97s" (UID: "32273522-7dd5-42c8-b2ce-fb3c46906000") : secret "image-registry-tls" not found Sep 11 14:15:21.748187 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.748160 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32273522-7dd5-42c8-b2ce-fb3c46906000-trusted-ca\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.748437 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.748189 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-bound-sa-token\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.748437 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.748214 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68438637-9ab0-4bfe-9074-767a01a364b5-config-volume\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:21.748437 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.748239 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqk57\" (UniqueName: \"kubernetes.io/projected/68438637-9ab0-4bfe-9074-767a01a364b5-kube-api-access-cqk57\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:21.748437 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.748367 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68438637-9ab0-4bfe-9074-767a01a364b5-tmp-dir\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:21.748437 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.748404 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32273522-7dd5-42c8-b2ce-fb3c46906000-ca-trust-extracted\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.748437 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.748434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wknfs\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-kube-api-access-wknfs\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.748722 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.748461 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f4f4176-30cd-40b6-bc5d-3f1c1d2cd3ff-cert\") pod \"ingress-canary-g5n7k\" (UID: \"4f4f4176-30cd-40b6-bc5d-3f1c1d2cd3ff\") " pod="openshift-ingress-canary/ingress-canary-g5n7k" Sep 11 14:15:21.748722 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.748508 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-certificates\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.748722 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.748537 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32273522-7dd5-42c8-b2ce-fb3c46906000-installation-pull-secrets\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.748722 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.748564 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32273522-7dd5-42c8-b2ce-fb3c46906000-image-registry-private-configuration\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.748722 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.748708 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68438637-9ab0-4bfe-9074-767a01a364b5-tmp-dir\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:21.748978 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.748807 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32273522-7dd5-42c8-b2ce-fb3c46906000-ca-trust-extracted\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.748978 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.748859 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/68438637-9ab0-4bfe-9074-767a01a364b5-config-volume\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:21.749137 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.749111 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-certificates\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.749207 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.749187 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32273522-7dd5-42c8-b2ce-fb3c46906000-trusted-ca\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.752849 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.752812 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32273522-7dd5-42c8-b2ce-fb3c46906000-image-registry-private-configuration\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.752964 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.752811 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32273522-7dd5-42c8-b2ce-fb3c46906000-installation-pull-secrets\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.753011 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.752988 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f4f4176-30cd-40b6-bc5d-3f1c1d2cd3ff-cert\") pod \"ingress-canary-g5n7k\" (UID: \"4f4f4176-30cd-40b6-bc5d-3f1c1d2cd3ff\") " pod="openshift-ingress-canary/ingress-canary-g5n7k" Sep 11 14:15:21.757957 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.757932 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-bound-sa-token\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.758309 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.758286 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqk57\" (UniqueName: \"kubernetes.io/projected/68438637-9ab0-4bfe-9074-767a01a364b5-kube-api-access-cqk57\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:21.758404 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.758369 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csmpw\" (UniqueName: \"kubernetes.io/projected/4f4f4176-30cd-40b6-bc5d-3f1c1d2cd3ff-kube-api-access-csmpw\") pod \"ingress-canary-g5n7k\" (UID: \"4f4f4176-30cd-40b6-bc5d-3f1c1d2cd3ff\") " pod="openshift-ingress-canary/ingress-canary-g5n7k" Sep 11 14:15:21.758404 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.758376 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wknfs\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-kube-api-access-wknfs\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:21.849646 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.849607 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsn77\" (UniqueName: \"kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77\") pod \"network-check-target-fh2vt\" (UID: \"65f1ece3-078e-4c34-9293-e2a4e6007ea8\") " pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:21.849812 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:21.849681 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:21.849812 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:21.849706 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:21.849812 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:21.849721 2573 projected.go:196] Error preparing data for projected volume kube-api-access-gsn77 for pod openshift-network-diagnostics/network-check-target-fh2vt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:21.849812 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.849760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs\") pod \"network-metrics-daemon-fh5nz\" (UID: \"afa6cfac-b37f-4e64-a308-658d8a3b62d9\") " pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:15:21.849812 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:21.849785 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77 podName:65f1ece3-078e-4c34-9293-e2a4e6007ea8 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:53.849769216 +0000 UTC m=+66.186737388 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gsn77" (UniqueName: "kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77") pod "network-check-target-fh2vt" (UID: "65f1ece3-078e-4c34-9293-e2a4e6007ea8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:21.850072 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:21.849868 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:21.850072 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:21.849940 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs podName:afa6cfac-b37f-4e64-a308-658d8a3b62d9 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:53.849922306 +0000 UTC m=+66.186890494 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs") pod "network-metrics-daemon-fh5nz" (UID: "afa6cfac-b37f-4e64-a308-658d8a3b62d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:21.877446 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.877421 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g5n7k" Sep 11 14:15:21.950652 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:21.950623 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret\") pod \"global-pull-secret-syncer-m8lns\" (UID: \"de580cd5-5351-479c-a9ad-0c0936042074\") " pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:21.951648 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:21.950824 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:21.951648 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:21.950983 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret podName:de580cd5-5351-479c-a9ad-0c0936042074 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:53.95096101 +0000 UTC m=+66.287929191 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret") pod "global-pull-secret-syncer-m8lns" (UID: "de580cd5-5351-479c-a9ad-0c0936042074") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:22.170039 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:22.169969 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g5n7k"] Sep 11 14:15:22.173346 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:15:22.173315 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4f4176_30cd_40b6_bc5d_3f1c1d2cd3ff.slice/crio-59fe259003ce80daa370dee4a5ff6e45638011cff9e1f6d50d98cfafc6986d35 WatchSource:0}: Error finding container 59fe259003ce80daa370dee4a5ff6e45638011cff9e1f6d50d98cfafc6986d35: Status 404 returned error can't find the container with id 59fe259003ce80daa370dee4a5ff6e45638011cff9e1f6d50d98cfafc6986d35 Sep 11 14:15:22.180556 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:22.180535 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:22.183289 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:22.183268 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 14:15:22.252685 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:22.252654 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:22.252853 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:22.252697 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:22.252853 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:22.252828 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:15:22.252853 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:22.252848 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:22.252997 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:22.252915 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls podName:68438637-9ab0-4bfe-9074-767a01a364b5 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:23.252898458 +0000 UTC m=+35.589866632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls") pod "dns-default-4gpp4" (UID: "68438637-9ab0-4bfe-9074-767a01a364b5") : secret "dns-default-metrics-tls" not found Sep 11 14:15:22.252997 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:22.252853 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-856c4b595b-4k97s: secret "image-registry-tls" not found Sep 11 14:15:22.252997 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:22.252983 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls podName:32273522-7dd5-42c8-b2ce-fb3c46906000 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:23.252971124 +0000 UTC m=+35.589939293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls") pod "image-registry-856c4b595b-4k97s" (UID: "32273522-7dd5-42c8-b2ce-fb3c46906000") : secret "image-registry-tls" not found Sep 11 14:15:22.335639 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:22.335603 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g5n7k" event={"ID":"4f4f4176-30cd-40b6-bc5d-3f1c1d2cd3ff","Type":"ContainerStarted","Data":"59fe259003ce80daa370dee4a5ff6e45638011cff9e1f6d50d98cfafc6986d35"} Sep 11 14:15:23.181283 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:23.181252 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:15:23.181467 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:23.181252 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:23.184592 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:23.184562 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 14:15:23.184981 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:23.184649 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 14:15:23.185741 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:23.185720 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vtjb2\"" Sep 11 14:15:23.185883 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:23.185804 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lc92k\"" Sep 11 14:15:23.186393 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:23.186373 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 14:15:23.261544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:23.261512 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:23.261692 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:23.261552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:23.261748 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:23.261696 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:23.261795 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:23.261761 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:15:23.261795 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:23.261768 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls podName:68438637-9ab0-4bfe-9074-767a01a364b5 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:25.261743281 +0000 UTC m=+37.598711452 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls") pod "dns-default-4gpp4" (UID: "68438637-9ab0-4bfe-9074-767a01a364b5") : secret "dns-default-metrics-tls" not found Sep 11 14:15:23.261795 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:23.261776 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-856c4b595b-4k97s: secret "image-registry-tls" not found Sep 11 14:15:23.261949 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:23.261827 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls podName:32273522-7dd5-42c8-b2ce-fb3c46906000 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:25.261812044 +0000 UTC m=+37.598780214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls") pod "image-registry-856c4b595b-4k97s" (UID: "32273522-7dd5-42c8-b2ce-fb3c46906000") : secret "image-registry-tls" not found Sep 11 14:15:23.554036 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:23.553845 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sn4pw_7d87c49f-6095-47cf-8cc2-736f36cb9a74/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:23.666679 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:23.666647 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sn4pw_7d87c49f-6095-47cf-8cc2-736f36cb9a74/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:24.237892 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:24.237863 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sn4pw_7d87c49f-6095-47cf-8cc2-736f36cb9a74/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:24.337993 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:24.337957 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sn4pw_7d87c49f-6095-47cf-8cc2-736f36cb9a74/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:25.277467 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:25.277212 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:25.277900 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:25.277492 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:25.277900 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:25.277395 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:25.277900 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:25.277613 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls podName:68438637-9ab0-4bfe-9074-767a01a364b5 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:29.277595219 +0000 UTC m=+41.614563393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls") pod "dns-default-4gpp4" (UID: "68438637-9ab0-4bfe-9074-767a01a364b5") : secret "dns-default-metrics-tls" not found Sep 11 14:15:25.277900 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:25.277682 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:15:25.277900 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:25.277696 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-856c4b595b-4k97s: secret "image-registry-tls" not found Sep 11 14:15:25.277900 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:25.277749 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls podName:32273522-7dd5-42c8-b2ce-fb3c46906000 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:29.277734785 +0000 UTC m=+41.614702955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls") pod "image-registry-856c4b595b-4k97s" (UID: "32273522-7dd5-42c8-b2ce-fb3c46906000") : secret "image-registry-tls" not found Sep 11 14:15:25.343498 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:25.343464 2573 generic.go:356] "Generic (PLEG): container finished" podID="eec501bb-6445-4360-8609-7f5a8f557a6a" containerID="4aa0c53dfa0e637066c0e35532e1c247d2db16c75995897c247c022aca782a36" exitCode=0 Sep 11 14:15:25.343631 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:25.343516 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk42w" event={"ID":"eec501bb-6445-4360-8609-7f5a8f557a6a","Type":"ContainerDied","Data":"4aa0c53dfa0e637066c0e35532e1c247d2db16c75995897c247c022aca782a36"} Sep 11 14:15:26.349065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:26.349027 2573 generic.go:356] "Generic (PLEG): container finished" podID="eec501bb-6445-4360-8609-7f5a8f557a6a" containerID="b654cf88bb48606d191afd93c3673dc47cb647c37f93a18990f472873116e616" exitCode=0 Sep 11 14:15:26.349467 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:26.349099 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk42w" event={"ID":"eec501bb-6445-4360-8609-7f5a8f557a6a","Type":"ContainerDied","Data":"b654cf88bb48606d191afd93c3673dc47cb647c37f93a18990f472873116e616"} Sep 11 14:15:26.389536 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:26.389507 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sn4pw_7d87c49f-6095-47cf-8cc2-736f36cb9a74/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:26.543639 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:26.543596 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sn4pw_7d87c49f-6095-47cf-8cc2-736f36cb9a74/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:26.613339 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:26.613283 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sn4pw_7d87c49f-6095-47cf-8cc2-736f36cb9a74/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:27.355305 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:27.355270 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jk42w" event={"ID":"eec501bb-6445-4360-8609-7f5a8f557a6a","Type":"ContainerStarted","Data":"c48bcf03f20a2b85c980e32d476279a25ebceb37a2a08a331dc8ca68f7ace43c"} Sep 11 14:15:27.356687 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:27.356666 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g5n7k" event={"ID":"4f4f4176-30cd-40b6-bc5d-3f1c1d2cd3ff","Type":"ContainerStarted","Data":"fef17cffb15211639a474c6e50204e739dc0877165502d21030f6f1c3c7a5812"} Sep 11 14:15:27.389654 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:27.389611 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jk42w" podStartSLOduration=9.06999496 podStartE2EDuration="39.389594326s" totalImagesPullingTime="30.319599366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:27.388383475 +0000 UTC m=+39.725351665" watchObservedRunningTime="2026-09-11 14:15:27.389594326 +0000 UTC m=+39.726562516" Sep 11 14:15:27.399104 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:27.399077 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sn4pw_7d87c49f-6095-47cf-8cc2-736f36cb9a74/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:27.435057 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:27.434993 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-g5n7k" podStartSLOduration=2.236141733 podStartE2EDuration="6.434973811s" totalImagesPullingTime="4.198832078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:27.433079448 +0000 UTC m=+39.770047639" watchObservedRunningTime="2026-09-11 14:15:27.434973811 +0000 UTC m=+39.771942006" Sep 11 14:15:27.514661 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:27.514626 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sn4pw_7d87c49f-6095-47cf-8cc2-736f36cb9a74/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:27.620171 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:27.620089 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sn4pw_7d87c49f-6095-47cf-8cc2-736f36cb9a74/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:27.722646 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:27.722614 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sn4pw_7d87c49f-6095-47cf-8cc2-736f36cb9a74/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:28.020664 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:28.020582 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sn4pw_7d87c49f-6095-47cf-8cc2-736f36cb9a74/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:28.125299 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:28.125269 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sn4pw_7d87c49f-6095-47cf-8cc2-736f36cb9a74/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:29.306751 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:29.306721 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:29.307104 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:29.306754 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:29.307104 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:29.306877 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:29.307104 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:29.306907 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:15:29.307104 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:29.306918 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-856c4b595b-4k97s: secret "image-registry-tls" not found Sep 11 14:15:29.307104 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:29.306955 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls podName:68438637-9ab0-4bfe-9074-767a01a364b5 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:37.306931867 +0000 UTC m=+49.643900036 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls") pod "dns-default-4gpp4" (UID: "68438637-9ab0-4bfe-9074-767a01a364b5") : secret "dns-default-metrics-tls" not found Sep 11 14:15:29.307104 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:29.306970 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls podName:32273522-7dd5-42c8-b2ce-fb3c46906000 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:37.306963879 +0000 UTC m=+49.643932047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls") pod "image-registry-856c4b595b-4k97s" (UID: "32273522-7dd5-42c8-b2ce-fb3c46906000") : secret "image-registry-tls" not found Sep 11 14:15:37.366944 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:37.366902 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:37.366944 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:37.366945 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:37.367413 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:37.367006 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:37.367413 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:37.367068 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:15:37.367413 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:37.367079 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-856c4b595b-4k97s: secret "image-registry-tls" not found Sep 11 14:15:37.367413 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:37.367079 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls podName:68438637-9ab0-4bfe-9074-767a01a364b5 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:53.367065092 +0000 UTC m=+65.704033262 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls") pod "dns-default-4gpp4" (UID: "68438637-9ab0-4bfe-9074-767a01a364b5") : secret "dns-default-metrics-tls" not found Sep 11 14:15:37.367413 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:37.367123 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls podName:32273522-7dd5-42c8-b2ce-fb3c46906000 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:53.367113114 +0000 UTC m=+65.704081284 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls") pod "image-registry-856c4b595b-4k97s" (UID: "32273522-7dd5-42c8-b2ce-fb3c46906000") : secret "image-registry-tls" not found Sep 11 14:15:48.332875 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:48.332841 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d87x6" Sep 11 14:15:53.376199 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:53.376163 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:15:53.376199 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:53.376197 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:15:53.376588 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:53.376317 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:53.376588 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:53.376337 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:15:53.376588 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:53.376348 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-856c4b595b-4k97s: secret "image-registry-tls" not found Sep 11 14:15:53.376588 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:53.376382 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls podName:68438637-9ab0-4bfe-9074-767a01a364b5 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:25.376367298 +0000 UTC m=+97.713335466 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls") pod "dns-default-4gpp4" (UID: "68438637-9ab0-4bfe-9074-767a01a364b5") : secret "dns-default-metrics-tls" not found Sep 11 14:15:53.376588 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:53.376396 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls podName:32273522-7dd5-42c8-b2ce-fb3c46906000 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:25.37639079 +0000 UTC m=+97.713358959 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls") pod "image-registry-856c4b595b-4k97s" (UID: "32273522-7dd5-42c8-b2ce-fb3c46906000") : secret "image-registry-tls" not found Sep 11 14:15:53.879210 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:53.879174 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs\") pod \"network-metrics-daemon-fh5nz\" (UID: \"afa6cfac-b37f-4e64-a308-658d8a3b62d9\") " pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:15:53.879403 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:53.879217 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsn77\" (UniqueName: \"kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77\") pod \"network-check-target-fh2vt\" (UID: \"65f1ece3-078e-4c34-9293-e2a4e6007ea8\") " pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:53.882635 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:53.882612 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 14:15:53.882773 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:53.882758 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 14:15:53.889781 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:53.889763 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 14:15:53.889904 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:15:53.889812 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs podName:afa6cfac-b37f-4e64-a308-658d8a3b62d9 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:57.889799219 +0000 UTC m=+130.226767388 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs") pod "network-metrics-daemon-fh5nz" (UID: "afa6cfac-b37f-4e64-a308-658d8a3b62d9") : secret "metrics-daemon-secret" not found Sep 11 14:15:53.893139 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:53.893121 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 14:15:53.903020 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:53.902989 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsn77\" (UniqueName: \"kubernetes.io/projected/65f1ece3-078e-4c34-9293-e2a4e6007ea8-kube-api-access-gsn77\") pod \"network-check-target-fh2vt\" (UID: \"65f1ece3-078e-4c34-9293-e2a4e6007ea8\") " pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:53.979502 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:53.979475 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret\") pod \"global-pull-secret-syncer-m8lns\" (UID: \"de580cd5-5351-479c-a9ad-0c0936042074\") " pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:53.986563 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:53.986548 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 14:15:53.992406 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:53.992379 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de580cd5-5351-479c-a9ad-0c0936042074-original-pull-secret\") pod \"global-pull-secret-syncer-m8lns\" (UID: \"de580cd5-5351-479c-a9ad-0c0936042074\") " pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:54.100823 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:54.100791 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lc92k\"" Sep 11 14:15:54.108384 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:54.108367 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:54.291181 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:54.291153 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m8lns" Sep 11 14:15:54.417817 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:54.417749 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fh2vt"] Sep 11 14:15:54.421740 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:15:54.421713 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f1ece3_078e_4c34_9293_e2a4e6007ea8.slice/crio-0fc9e3de8f82ece8112741acd0da61c478dcd6dd1b48f85b673cbf7661842230 WatchSource:0}: Error finding container 0fc9e3de8f82ece8112741acd0da61c478dcd6dd1b48f85b673cbf7661842230: Status 404 returned error can't find the container with id 0fc9e3de8f82ece8112741acd0da61c478dcd6dd1b48f85b673cbf7661842230 Sep 11 14:15:54.594285 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:54.594256 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m8lns"] Sep 11 14:15:54.597514 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:15:54.597487 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde580cd5_5351_479c_a9ad_0c0936042074.slice/crio-22587154b4961e89438df34ea793e56ba4d2250f4f6e2f61b45a9761a5186c0d WatchSource:0}: Error finding container 22587154b4961e89438df34ea793e56ba4d2250f4f6e2f61b45a9761a5186c0d: Status 404 returned error can't find the container with id 22587154b4961e89438df34ea793e56ba4d2250f4f6e2f61b45a9761a5186c0d Sep 11 14:15:55.421307 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:55.421268 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m8lns" event={"ID":"de580cd5-5351-479c-a9ad-0c0936042074","Type":"ContainerStarted","Data":"22587154b4961e89438df34ea793e56ba4d2250f4f6e2f61b45a9761a5186c0d"} Sep 11 14:15:55.422935 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:55.422888 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fh2vt" event={"ID":"65f1ece3-078e-4c34-9293-e2a4e6007ea8","Type":"ContainerStarted","Data":"0fc9e3de8f82ece8112741acd0da61c478dcd6dd1b48f85b673cbf7661842230"} Sep 11 14:15:59.433117 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:59.433078 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m8lns" event={"ID":"de580cd5-5351-479c-a9ad-0c0936042074","Type":"ContainerStarted","Data":"8ee9f7cad9a60b160557a5997f34179cd1892e2808e0bda0eb55a60e46c343e0"} Sep 11 14:15:59.434497 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:59.434472 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fh2vt" event={"ID":"65f1ece3-078e-4c34-9293-e2a4e6007ea8","Type":"ContainerStarted","Data":"9522ab8917fdf6b0ea6a7e9cee1ac2f7f7a9eeef0eafd86b7f58e8ee2d671212"} Sep 11 14:15:59.434636 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:59.434615 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:15:59.456007 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:59.455963 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m8lns" podStartSLOduration=67.120071602 podStartE2EDuration="1m11.455947754s" totalImagesPullingTime="4.335876152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:59.454158964 +0000 UTC m=+71.791127154" watchObservedRunningTime="2026-09-11 14:15:59.455947754 +0000 UTC m=+71.792915942" Sep 11 14:15:59.479965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:15:59.479922 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fh2vt" podStartSLOduration=66.970839684 podStartE2EDuration="1m11.479910191s" totalImagesPullingTime="4.509070507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:59.479572957 +0000 UTC m=+71.816541158" watchObservedRunningTime="2026-09-11 14:15:59.479910191 +0000 UTC m=+71.816878372" Sep 11 14:16:01.679682 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.679646 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z"] Sep 11 14:16:01.683499 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.683479 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z" Sep 11 14:16:01.700040 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.700020 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 11 14:16:01.701993 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.701976 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z"] Sep 11 14:16:01.704905 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.704886 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 11 14:16:01.708753 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.708733 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 11 14:16:01.712565 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.712548 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7q84k\"" Sep 11 14:16:01.713878 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.713862 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:16:01.733822 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.733800 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh7j8\" (UniqueName: \"kubernetes.io/projected/653a4ec0-e4b5-4956-9fe8-969d4eb6a093-kube-api-access-gh7j8\") pod \"service-ca-operator-cf7c776b7-4g66z\" (UID: \"653a4ec0-e4b5-4956-9fe8-969d4eb6a093\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z" Sep 11 14:16:01.733915 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.733843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/653a4ec0-e4b5-4956-9fe8-969d4eb6a093-serving-cert\") pod \"service-ca-operator-cf7c776b7-4g66z\" (UID: \"653a4ec0-e4b5-4956-9fe8-969d4eb6a093\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z" Sep 11 14:16:01.733967 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.733932 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/653a4ec0-e4b5-4956-9fe8-969d4eb6a093-config\") pod \"service-ca-operator-cf7c776b7-4g66z\" (UID: \"653a4ec0-e4b5-4956-9fe8-969d4eb6a093\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z" Sep 11 14:16:01.834938 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.834912 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/653a4ec0-e4b5-4956-9fe8-969d4eb6a093-config\") pod \"service-ca-operator-cf7c776b7-4g66z\" (UID: \"653a4ec0-e4b5-4956-9fe8-969d4eb6a093\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z" Sep 11 14:16:01.835084 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.834961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh7j8\" (UniqueName: \"kubernetes.io/projected/653a4ec0-e4b5-4956-9fe8-969d4eb6a093-kube-api-access-gh7j8\") pod \"service-ca-operator-cf7c776b7-4g66z\" (UID: \"653a4ec0-e4b5-4956-9fe8-969d4eb6a093\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z" Sep 11 14:16:01.835084 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.834989 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/653a4ec0-e4b5-4956-9fe8-969d4eb6a093-serving-cert\") pod \"service-ca-operator-cf7c776b7-4g66z\" (UID: \"653a4ec0-e4b5-4956-9fe8-969d4eb6a093\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z" Sep 11 14:16:01.835531 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.835511 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/653a4ec0-e4b5-4956-9fe8-969d4eb6a093-config\") pod \"service-ca-operator-cf7c776b7-4g66z\" (UID: \"653a4ec0-e4b5-4956-9fe8-969d4eb6a093\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z" Sep 11 14:16:01.837143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.837123 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/653a4ec0-e4b5-4956-9fe8-969d4eb6a093-serving-cert\") pod \"service-ca-operator-cf7c776b7-4g66z\" (UID: \"653a4ec0-e4b5-4956-9fe8-969d4eb6a093\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z" Sep 11 14:16:01.848040 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.848022 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh7j8\" (UniqueName: \"kubernetes.io/projected/653a4ec0-e4b5-4956-9fe8-969d4eb6a093-kube-api-access-gh7j8\") pod \"service-ca-operator-cf7c776b7-4g66z\" (UID: \"653a4ec0-e4b5-4956-9fe8-969d4eb6a093\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z" Sep 11 14:16:01.990981 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:01.990897 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z" Sep 11 14:16:02.259670 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:02.259562 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z"] Sep 11 14:16:02.263466 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:16:02.263435 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653a4ec0_e4b5_4956_9fe8_969d4eb6a093.slice/crio-c79a5177163c4fec6cf9a7d56701302a9ef721058baa5deedf1a4e813f6c9f56 WatchSource:0}: Error finding container c79a5177163c4fec6cf9a7d56701302a9ef721058baa5deedf1a4e813f6c9f56: Status 404 returned error can't find the container with id c79a5177163c4fec6cf9a7d56701302a9ef721058baa5deedf1a4e813f6c9f56 Sep 11 14:16:02.446262 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:02.446229 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z" event={"ID":"653a4ec0-e4b5-4956-9fe8-969d4eb6a093","Type":"ContainerStarted","Data":"c79a5177163c4fec6cf9a7d56701302a9ef721058baa5deedf1a4e813f6c9f56"} Sep 11 14:16:04.453891 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:04.453794 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z" event={"ID":"653a4ec0-e4b5-4956-9fe8-969d4eb6a093","Type":"ContainerStarted","Data":"c02fbbc0b820aa625e2c67e7fc4a368fb79d7b7deaf220f6c2147d82d43a3352"} Sep 11 14:16:04.471260 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:04.471213 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z" podStartSLOduration=1.569032491 podStartE2EDuration="3.471197876s" totalImagesPullingTime="1.902165385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:04.470429627 +0000 UTC m=+76.807397817" watchObservedRunningTime="2026-09-11 14:16:04.471197876 +0000 UTC m=+76.808166066" Sep 11 14:16:06.532288 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.532253 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tvmcz"] Sep 11 14:16:06.535443 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.535416 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:06.538823 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.538799 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 14:16:06.538950 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.538799 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 14:16:06.538950 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.538852 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 14:16:06.538950 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.538889 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 14:16:06.540325 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.540308 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6n4gf\"" Sep 11 14:16:06.558369 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.558344 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tvmcz"] Sep 11 14:16:06.572330 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.572307 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eb988f1-4b9c-44db-85c6-5a817446c56d-crio-socket\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:06.572422 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.572339 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtqmf\" (UniqueName: \"kubernetes.io/projected/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-api-access-dtqmf\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:06.572422 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.572358 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eb988f1-4b9c-44db-85c6-5a817446c56d-data-volume\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:06.572422 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.572377 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:06.572522 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.572491 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:06.673019 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.672992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eb988f1-4b9c-44db-85c6-5a817446c56d-crio-socket\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:06.673118 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.673030 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtqmf\" (UniqueName: \"kubernetes.io/projected/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-api-access-dtqmf\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:06.673118 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.673049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eb988f1-4b9c-44db-85c6-5a817446c56d-data-volume\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:06.673118 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.673071 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:06.673233 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.673111 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eb988f1-4b9c-44db-85c6-5a817446c56d-crio-socket\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:06.673233 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:06.673168 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:06.673233 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.673190 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:06.673233 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:06.673218 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls podName:6eb988f1-4b9c-44db-85c6-5a817446c56d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:07.173205585 +0000 UTC m=+79.510173753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvmcz" (UID: "6eb988f1-4b9c-44db-85c6-5a817446c56d") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:06.673375 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:06.673247 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:06.673375 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:06.673321 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm podName:6eb988f1-4b9c-44db-85c6-5a817446c56d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:07.173298813 +0000 UTC m=+79.510267001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvmcz" (UID: "6eb988f1-4b9c-44db-85c6-5a817446c56d") : configmap "kube-rbac-proxy" not found Sep 11 14:16:06.673439 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.673390 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eb988f1-4b9c-44db-85c6-5a817446c56d-data-volume\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:06.683497 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:06.683471 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtqmf\" (UniqueName: \"kubernetes.io/projected/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-api-access-dtqmf\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:07.176810 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:07.176775 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:07.176977 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:07.176827 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:07.176977 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:07.176896 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:07.176977 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:07.176937 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:07.176977 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:07.176960 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm podName:6eb988f1-4b9c-44db-85c6-5a817446c56d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:08.176945186 +0000 UTC m=+80.513913355 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvmcz" (UID: "6eb988f1-4b9c-44db-85c6-5a817446c56d") : configmap "kube-rbac-proxy" not found Sep 11 14:16:07.177122 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:07.176992 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls podName:6eb988f1-4b9c-44db-85c6-5a817446c56d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:08.176980433 +0000 UTC m=+80.513948607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvmcz" (UID: "6eb988f1-4b9c-44db-85c6-5a817446c56d") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:08.183314 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.183284 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:08.183726 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.183332 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:08.183726 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:08.183428 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:08.183726 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:08.183446 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:08.183726 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:08.183490 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm podName:6eb988f1-4b9c-44db-85c6-5a817446c56d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:10.183475211 +0000 UTC m=+82.520443380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvmcz" (UID: "6eb988f1-4b9c-44db-85c6-5a817446c56d") : configmap "kube-rbac-proxy" not found Sep 11 14:16:08.183726 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:08.183504 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls podName:6eb988f1-4b9c-44db-85c6-5a817446c56d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:10.183497995 +0000 UTC m=+82.520466164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvmcz" (UID: "6eb988f1-4b9c-44db-85c6-5a817446c56d") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:08.457852 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.457762 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l7mvd_ee1912ca-8cb0-4013-bc95-5cb2210bbfe3/dns-node-resolver/0.log" Sep 11 14:16:08.461055 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.461033 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-h59sl"] Sep 11 14:16:08.463943 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.463925 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-h59sl" Sep 11 14:16:08.466759 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.466743 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 11 14:16:08.466846 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.466820 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 11 14:16:08.467935 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.467909 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 11 14:16:08.468012 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.467950 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nqsxr\"" Sep 11 14:16:08.468012 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.467929 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 11 14:16:08.476134 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.476114 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-h59sl"] Sep 11 14:16:08.485774 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.485756 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74kg7\" (UniqueName: \"kubernetes.io/projected/1ce9e19d-f72a-4a06-aaa5-62db675de751-kube-api-access-74kg7\") pod \"service-ca-568869fdc5-h59sl\" (UID: \"1ce9e19d-f72a-4a06-aaa5-62db675de751\") " pod="openshift-service-ca/service-ca-568869fdc5-h59sl" Sep 11 14:16:08.485854 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.485791 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1ce9e19d-f72a-4a06-aaa5-62db675de751-signing-cabundle\") pod \"service-ca-568869fdc5-h59sl\" (UID: \"1ce9e19d-f72a-4a06-aaa5-62db675de751\") " pod="openshift-service-ca/service-ca-568869fdc5-h59sl" Sep 11 14:16:08.485854 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.485819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1ce9e19d-f72a-4a06-aaa5-62db675de751-signing-key\") pod \"service-ca-568869fdc5-h59sl\" (UID: \"1ce9e19d-f72a-4a06-aaa5-62db675de751\") " pod="openshift-service-ca/service-ca-568869fdc5-h59sl" Sep 11 14:16:08.586472 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.586442 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1ce9e19d-f72a-4a06-aaa5-62db675de751-signing-cabundle\") pod \"service-ca-568869fdc5-h59sl\" (UID: \"1ce9e19d-f72a-4a06-aaa5-62db675de751\") " pod="openshift-service-ca/service-ca-568869fdc5-h59sl" Sep 11 14:16:08.586586 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.586481 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1ce9e19d-f72a-4a06-aaa5-62db675de751-signing-key\") pod \"service-ca-568869fdc5-h59sl\" (UID: \"1ce9e19d-f72a-4a06-aaa5-62db675de751\") " pod="openshift-service-ca/service-ca-568869fdc5-h59sl" Sep 11 14:16:08.586586 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.586545 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74kg7\" (UniqueName: \"kubernetes.io/projected/1ce9e19d-f72a-4a06-aaa5-62db675de751-kube-api-access-74kg7\") pod \"service-ca-568869fdc5-h59sl\" (UID: \"1ce9e19d-f72a-4a06-aaa5-62db675de751\") " pod="openshift-service-ca/service-ca-568869fdc5-h59sl" Sep 11 14:16:08.587069 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.587051 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1ce9e19d-f72a-4a06-aaa5-62db675de751-signing-cabundle\") pod \"service-ca-568869fdc5-h59sl\" (UID: \"1ce9e19d-f72a-4a06-aaa5-62db675de751\") " pod="openshift-service-ca/service-ca-568869fdc5-h59sl" Sep 11 14:16:08.588771 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.588753 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1ce9e19d-f72a-4a06-aaa5-62db675de751-signing-key\") pod \"service-ca-568869fdc5-h59sl\" (UID: \"1ce9e19d-f72a-4a06-aaa5-62db675de751\") " pod="openshift-service-ca/service-ca-568869fdc5-h59sl" Sep 11 14:16:08.600167 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.600144 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74kg7\" (UniqueName: \"kubernetes.io/projected/1ce9e19d-f72a-4a06-aaa5-62db675de751-kube-api-access-74kg7\") pod \"service-ca-568869fdc5-h59sl\" (UID: \"1ce9e19d-f72a-4a06-aaa5-62db675de751\") " pod="openshift-service-ca/service-ca-568869fdc5-h59sl" Sep 11 14:16:08.776088 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:08.776025 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-h59sl" Sep 11 14:16:09.052541 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:09.052474 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-h59sl"] Sep 11 14:16:09.057203 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:16:09.055428 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce9e19d_f72a_4a06_aaa5_62db675de751.slice/crio-5350c1c8026291b794ba8c1ced4eb20578fc3b75e2e188585b76bd7956e89297 WatchSource:0}: Error finding container 5350c1c8026291b794ba8c1ced4eb20578fc3b75e2e188585b76bd7956e89297: Status 404 returned error can't find the container with id 5350c1c8026291b794ba8c1ced4eb20578fc3b75e2e188585b76bd7956e89297 Sep 11 14:16:09.468411 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:09.468374 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-h59sl" event={"ID":"1ce9e19d-f72a-4a06-aaa5-62db675de751","Type":"ContainerStarted","Data":"b00c55cf707c8a146fdcbb20a3ac4010e0314315812cf82e290a1ad1089a781a"} Sep 11 14:16:09.468785 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:09.468424 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-h59sl" event={"ID":"1ce9e19d-f72a-4a06-aaa5-62db675de751","Type":"ContainerStarted","Data":"5350c1c8026291b794ba8c1ced4eb20578fc3b75e2e188585b76bd7956e89297"} Sep 11 14:16:09.494950 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:09.494893 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-h59sl" podStartSLOduration=1.494875644 podStartE2EDuration="1.494875644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:09.49446926 +0000 UTC m=+81.831437452" watchObservedRunningTime="2026-09-11 14:16:09.494875644 +0000 UTC m=+81.831843837" Sep 11 14:16:09.656535 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:09.656508 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pb9cz_9c142fa3-73aa-434a-a3c8-9d3694c03001/node-ca/0.log" Sep 11 14:16:10.058241 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:10.058196 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g5n7k_4f4f4176-30cd-40b6-bc5d-3f1c1d2cd3ff/serve-healthcheck-canary/0.log" Sep 11 14:16:10.198101 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:10.198061 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:10.198308 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:10.198132 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:10.198308 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:10.198200 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:10.198308 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:10.198229 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:10.198308 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:10.198277 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm podName:6eb988f1-4b9c-44db-85c6-5a817446c56d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:14.198256182 +0000 UTC m=+86.535224366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvmcz" (UID: "6eb988f1-4b9c-44db-85c6-5a817446c56d") : configmap "kube-rbac-proxy" not found Sep 11 14:16:10.198308 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:10.198296 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls podName:6eb988f1-4b9c-44db-85c6-5a817446c56d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:14.198287223 +0000 UTC m=+86.535255395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvmcz" (UID: "6eb988f1-4b9c-44db-85c6-5a817446c56d") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:14.231542 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:14.231422 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:14.231542 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:14.231492 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:14.232011 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:14.231546 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:14.232011 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:14.231617 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm podName:6eb988f1-4b9c-44db-85c6-5a817446c56d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:22.231599181 +0000 UTC m=+94.568567365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tvmcz" (UID: "6eb988f1-4b9c-44db-85c6-5a817446c56d") : configmap "kube-rbac-proxy" not found Sep 11 14:16:14.232011 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:14.231623 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:14.232011 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:14.231659 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls podName:6eb988f1-4b9c-44db-85c6-5a817446c56d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:22.231648314 +0000 UTC m=+94.568616496 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tvmcz" (UID: "6eb988f1-4b9c-44db-85c6-5a817446c56d") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:22.294145 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:22.294095 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:22.294527 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:22.294177 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:22.294623 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:22.294605 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eb988f1-4b9c-44db-85c6-5a817446c56d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:22.296375 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:22.296353 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eb988f1-4b9c-44db-85c6-5a817446c56d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tvmcz\" (UID: \"6eb988f1-4b9c-44db-85c6-5a817446c56d\") " pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:22.444640 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:22.444610 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tvmcz" Sep 11 14:16:22.718770 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:22.718742 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tvmcz"] Sep 11 14:16:22.721586 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:16:22.721553 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb988f1_4b9c_44db_85c6_5a817446c56d.slice/crio-d8b9f8f8bd54dfacab46d2cd76ff47b3493a255caa58061c90aaa6d9d72143a4 WatchSource:0}: Error finding container d8b9f8f8bd54dfacab46d2cd76ff47b3493a255caa58061c90aaa6d9d72143a4: Status 404 returned error can't find the container with id d8b9f8f8bd54dfacab46d2cd76ff47b3493a255caa58061c90aaa6d9d72143a4 Sep 11 14:16:23.507170 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:23.507138 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tvmcz" event={"ID":"6eb988f1-4b9c-44db-85c6-5a817446c56d","Type":"ContainerStarted","Data":"d7d665d09740db0d88601ccaca3f8f3b9db2f2e9c388fddc426e3cd1635b2872"} Sep 11 14:16:23.507509 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:23.507191 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tvmcz" event={"ID":"6eb988f1-4b9c-44db-85c6-5a817446c56d","Type":"ContainerStarted","Data":"874a4f64914ef425e23f741a6380c42b3b376bd14d21f703394b3bbebd3203c7"} Sep 11 14:16:23.507509 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:23.507210 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tvmcz" event={"ID":"6eb988f1-4b9c-44db-85c6-5a817446c56d","Type":"ContainerStarted","Data":"d8b9f8f8bd54dfacab46d2cd76ff47b3493a255caa58061c90aaa6d9d72143a4"} Sep 11 14:16:25.416340 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:25.416301 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:16:25.416340 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:25.416343 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:16:25.418671 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:25.418640 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/68438637-9ab0-4bfe-9074-767a01a364b5-metrics-tls\") pod \"dns-default-4gpp4\" (UID: \"68438637-9ab0-4bfe-9074-767a01a364b5\") " pod="openshift-dns/dns-default-4gpp4" Sep 11 14:16:25.418785 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:25.418760 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls\") pod \"image-registry-856c4b595b-4k97s\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:16:25.455958 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:25.455934 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jb4sz\"" Sep 11 14:16:25.462345 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:25.462328 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:16:25.494913 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:25.494884 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xblvp\"" Sep 11 14:16:25.505328 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:25.505297 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4gpp4" Sep 11 14:16:25.518368 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:25.518338 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tvmcz" event={"ID":"6eb988f1-4b9c-44db-85c6-5a817446c56d","Type":"ContainerStarted","Data":"112fef22b780ed8c05c9610713aaca0c37676904628198d47ce39998830d76bc"} Sep 11 14:16:25.558744 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:25.558238 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tvmcz" podStartSLOduration=17.816523966 podStartE2EDuration="19.558219083s" totalImagesPullingTime="1.741695117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:16:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:25.555954262 +0000 UTC m=+97.892922454" watchObservedRunningTime="2026-09-11 14:16:25.558219083 +0000 UTC m=+97.895187274" Sep 11 14:16:25.755020 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:25.754947 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-856c4b595b-4k97s"] Sep 11 14:16:25.758766 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:16:25.758735 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32273522_7dd5_42c8_b2ce_fb3c46906000.slice/crio-5a71cd12e1e75e01025e2d73495a7bff00a4dc5cbd1e49f981c7ce41e2e078d9 WatchSource:0}: Error finding container 5a71cd12e1e75e01025e2d73495a7bff00a4dc5cbd1e49f981c7ce41e2e078d9: Status 404 returned error can't find the container with id 5a71cd12e1e75e01025e2d73495a7bff00a4dc5cbd1e49f981c7ce41e2e078d9 Sep 11 14:16:25.785940 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:25.785916 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4gpp4"] Sep 11 14:16:25.790782 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:16:25.790748 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68438637_9ab0_4bfe_9074_767a01a364b5.slice/crio-11293b671714781b66fd8eb9e4c5f7b6a48a3d067718902f1f4427a428d85458 WatchSource:0}: Error finding container 11293b671714781b66fd8eb9e4c5f7b6a48a3d067718902f1f4427a428d85458: Status 404 returned error can't find the container with id 11293b671714781b66fd8eb9e4c5f7b6a48a3d067718902f1f4427a428d85458 Sep 11 14:16:26.523760 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:26.523690 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4gpp4" event={"ID":"68438637-9ab0-4bfe-9074-767a01a364b5","Type":"ContainerStarted","Data":"11293b671714781b66fd8eb9e4c5f7b6a48a3d067718902f1f4427a428d85458"} Sep 11 14:16:26.525505 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:26.525476 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-856c4b595b-4k97s" event={"ID":"32273522-7dd5-42c8-b2ce-fb3c46906000","Type":"ContainerStarted","Data":"a4b0665fd4e2df8a70b62645b517c603927b68e23fc03ffcd539a76aeb2470e1"} Sep 11 14:16:26.525635 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:26.525527 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-856c4b595b-4k97s" event={"ID":"32273522-7dd5-42c8-b2ce-fb3c46906000","Type":"ContainerStarted","Data":"5a71cd12e1e75e01025e2d73495a7bff00a4dc5cbd1e49f981c7ce41e2e078d9"} Sep 11 14:16:26.576114 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:26.576068 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-856c4b595b-4k97s" podStartSLOduration=77.576050316 podStartE2EDuration="1m17.576050316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:26.574948124 +0000 UTC m=+98.911916316" watchObservedRunningTime="2026-09-11 14:16:26.576050316 +0000 UTC m=+98.913018506" Sep 11 14:16:27.530258 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:27.530218 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:16:28.534727 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:28.534690 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4gpp4" event={"ID":"68438637-9ab0-4bfe-9074-767a01a364b5","Type":"ContainerStarted","Data":"191d8833a76a90395e214f3a26c511748b9b33a229ff6ea099315c0ee3d3f7d1"} Sep 11 14:16:28.535137 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:28.534746 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4gpp4" event={"ID":"68438637-9ab0-4bfe-9074-767a01a364b5","Type":"ContainerStarted","Data":"58f630b3fa2b26f05b131cc002bd78a95a40a09026f29bbc308f2e89572a8cb3"} Sep 11 14:16:28.568626 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:28.568580 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4gpp4" podStartSLOduration=65.820304615 podStartE2EDuration="1m7.568565722s" totalImagesPullingTime="1.748261107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:28.568028652 +0000 UTC m=+100.904996843" watchObservedRunningTime="2026-09-11 14:16:28.568565722 +0000 UTC m=+100.905533910" Sep 11 14:16:29.537847 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:29.537807 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4gpp4" Sep 11 14:16:30.352267 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.352229 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj"] Sep 11 14:16:30.355195 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.355179 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj" Sep 11 14:16:30.368254 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.368230 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 11 14:16:30.368352 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.368257 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 14:16:30.368352 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.368272 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 14:16:30.371773 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.371753 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 14:16:30.378047 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.378027 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj"] Sep 11 14:16:30.442110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.442087 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fh2vt" Sep 11 14:16:30.454712 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.454686 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3aa047f7-a959-4f95-a04d-bdd9a415c65d-tmp\") pod \"klusterlet-addon-workmgr-dc456b57c-k7jtj\" (UID: \"3aa047f7-a959-4f95-a04d-bdd9a415c65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj" Sep 11 14:16:30.454815 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.454718 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3aa047f7-a959-4f95-a04d-bdd9a415c65d-klusterlet-config\") pod \"klusterlet-addon-workmgr-dc456b57c-k7jtj\" (UID: \"3aa047f7-a959-4f95-a04d-bdd9a415c65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj" Sep 11 14:16:30.454815 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.454751 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9qtc\" (UniqueName: \"kubernetes.io/projected/3aa047f7-a959-4f95-a04d-bdd9a415c65d-kube-api-access-c9qtc\") pod \"klusterlet-addon-workmgr-dc456b57c-k7jtj\" (UID: \"3aa047f7-a959-4f95-a04d-bdd9a415c65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj" Sep 11 14:16:30.461061 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.461041 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-krm4c"] Sep 11 14:16:30.463776 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.463760 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-krm4c" Sep 11 14:16:30.468605 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.468586 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 11 14:16:30.468882 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.468864 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 11 14:16:30.473847 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.473806 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-pwvrc\"" Sep 11 14:16:30.486695 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.486674 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-krm4c"] Sep 11 14:16:30.555185 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.555157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8bd8da52-9957-4092-b43d-d3f75992e3b5-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-krm4c\" (UID: \"8bd8da52-9957-4092-b43d-d3f75992e3b5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-krm4c" Sep 11 14:16:30.555684 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.555207 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9qtc\" (UniqueName: \"kubernetes.io/projected/3aa047f7-a959-4f95-a04d-bdd9a415c65d-kube-api-access-c9qtc\") pod \"klusterlet-addon-workmgr-dc456b57c-k7jtj\" (UID: \"3aa047f7-a959-4f95-a04d-bdd9a415c65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj" Sep 11 14:16:30.555684 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.555295 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8bd8da52-9957-4092-b43d-d3f75992e3b5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-krm4c\" (UID: \"8bd8da52-9957-4092-b43d-d3f75992e3b5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-krm4c" Sep 11 14:16:30.555684 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.555341 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3aa047f7-a959-4f95-a04d-bdd9a415c65d-tmp\") pod \"klusterlet-addon-workmgr-dc456b57c-k7jtj\" (UID: \"3aa047f7-a959-4f95-a04d-bdd9a415c65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj" Sep 11 14:16:30.555684 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.555405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3aa047f7-a959-4f95-a04d-bdd9a415c65d-klusterlet-config\") pod \"klusterlet-addon-workmgr-dc456b57c-k7jtj\" (UID: \"3aa047f7-a959-4f95-a04d-bdd9a415c65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj" Sep 11 14:16:30.555930 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.555735 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3aa047f7-a959-4f95-a04d-bdd9a415c65d-tmp\") pod \"klusterlet-addon-workmgr-dc456b57c-k7jtj\" (UID: \"3aa047f7-a959-4f95-a04d-bdd9a415c65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj" Sep 11 14:16:30.558308 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.558283 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3aa047f7-a959-4f95-a04d-bdd9a415c65d-klusterlet-config\") pod \"klusterlet-addon-workmgr-dc456b57c-k7jtj\" (UID: \"3aa047f7-a959-4f95-a04d-bdd9a415c65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj" Sep 11 14:16:30.586021 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.585996 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9qtc\" (UniqueName: \"kubernetes.io/projected/3aa047f7-a959-4f95-a04d-bdd9a415c65d-kube-api-access-c9qtc\") pod \"klusterlet-addon-workmgr-dc456b57c-k7jtj\" (UID: \"3aa047f7-a959-4f95-a04d-bdd9a415c65d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj" Sep 11 14:16:30.656106 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.656042 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8bd8da52-9957-4092-b43d-d3f75992e3b5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-krm4c\" (UID: \"8bd8da52-9957-4092-b43d-d3f75992e3b5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-krm4c" Sep 11 14:16:30.656238 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.656147 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8bd8da52-9957-4092-b43d-d3f75992e3b5-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-krm4c\" (UID: \"8bd8da52-9957-4092-b43d-d3f75992e3b5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-krm4c" Sep 11 14:16:30.656735 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.656716 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8bd8da52-9957-4092-b43d-d3f75992e3b5-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-krm4c\" (UID: \"8bd8da52-9957-4092-b43d-d3f75992e3b5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-krm4c" Sep 11 14:16:30.658342 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.658321 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8bd8da52-9957-4092-b43d-d3f75992e3b5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-krm4c\" (UID: \"8bd8da52-9957-4092-b43d-d3f75992e3b5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-krm4c" Sep 11 14:16:30.664113 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.664092 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj" Sep 11 14:16:30.771584 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.771560 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-krm4c" Sep 11 14:16:30.944513 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:30.944449 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj"] Sep 11 14:16:30.947236 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:16:30.947207 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa047f7_a959_4f95_a04d_bdd9a415c65d.slice/crio-4b637b6fce109ec9210618198fe5c21c2ae4bf7adf70fd9275f14f0bf4f80a22 WatchSource:0}: Error finding container 4b637b6fce109ec9210618198fe5c21c2ae4bf7adf70fd9275f14f0bf4f80a22: Status 404 returned error can't find the container with id 4b637b6fce109ec9210618198fe5c21c2ae4bf7adf70fd9275f14f0bf4f80a22 Sep 11 14:16:31.043245 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:31.043214 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-krm4c"] Sep 11 14:16:31.045872 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:16:31.045849 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd8da52_9957_4092_b43d_d3f75992e3b5.slice/crio-e06b1cbe3696f0c59b09440129a2d1a8485026ccb00548835ad367e206ae95bd WatchSource:0}: Error finding container e06b1cbe3696f0c59b09440129a2d1a8485026ccb00548835ad367e206ae95bd: Status 404 returned error can't find the container with id e06b1cbe3696f0c59b09440129a2d1a8485026ccb00548835ad367e206ae95bd Sep 11 14:16:31.545040 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:31.545000 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-krm4c" event={"ID":"8bd8da52-9957-4092-b43d-d3f75992e3b5","Type":"ContainerStarted","Data":"e06b1cbe3696f0c59b09440129a2d1a8485026ccb00548835ad367e206ae95bd"} Sep 11 14:16:31.546025 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:31.545992 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj" event={"ID":"3aa047f7-a959-4f95-a04d-bdd9a415c65d","Type":"ContainerStarted","Data":"4b637b6fce109ec9210618198fe5c21c2ae4bf7adf70fd9275f14f0bf4f80a22"} Sep 11 14:16:33.347330 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.347295 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-jw6px"] Sep 11 14:16:33.377256 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.377177 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-jw6px"] Sep 11 14:16:33.377406 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.377291 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" Sep 11 14:16:33.380758 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.380734 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 14:16:33.381959 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.381934 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 11 14:16:33.382065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.381935 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-d94l4\"" Sep 11 14:16:33.382148 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.382069 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 14:16:33.382278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.382254 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 11 14:16:33.382630 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.382610 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 14:16:33.482965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.482857 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r62zn\" (UniqueName: \"kubernetes.io/projected/0bcc91a3-c35b-4498-8a55-c67ee9c5662e-kube-api-access-r62zn\") pod \"prometheus-operator-68bcd7f987-jw6px\" (UID: \"0bcc91a3-c35b-4498-8a55-c67ee9c5662e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" Sep 11 14:16:33.482965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.482922 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0bcc91a3-c35b-4498-8a55-c67ee9c5662e-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-jw6px\" (UID: \"0bcc91a3-c35b-4498-8a55-c67ee9c5662e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" Sep 11 14:16:33.483199 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.483016 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0bcc91a3-c35b-4498-8a55-c67ee9c5662e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-jw6px\" (UID: \"0bcc91a3-c35b-4498-8a55-c67ee9c5662e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" Sep 11 14:16:33.483199 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.483059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bcc91a3-c35b-4498-8a55-c67ee9c5662e-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-jw6px\" (UID: \"0bcc91a3-c35b-4498-8a55-c67ee9c5662e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" Sep 11 14:16:33.584091 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.584053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r62zn\" (UniqueName: \"kubernetes.io/projected/0bcc91a3-c35b-4498-8a55-c67ee9c5662e-kube-api-access-r62zn\") pod \"prometheus-operator-68bcd7f987-jw6px\" (UID: \"0bcc91a3-c35b-4498-8a55-c67ee9c5662e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" Sep 11 14:16:33.584309 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.584103 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0bcc91a3-c35b-4498-8a55-c67ee9c5662e-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-jw6px\" (UID: \"0bcc91a3-c35b-4498-8a55-c67ee9c5662e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" Sep 11 14:16:33.584309 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.584139 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0bcc91a3-c35b-4498-8a55-c67ee9c5662e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-jw6px\" (UID: \"0bcc91a3-c35b-4498-8a55-c67ee9c5662e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" Sep 11 14:16:33.584309 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.584169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bcc91a3-c35b-4498-8a55-c67ee9c5662e-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-jw6px\" (UID: \"0bcc91a3-c35b-4498-8a55-c67ee9c5662e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" Sep 11 14:16:33.584872 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.584823 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0bcc91a3-c35b-4498-8a55-c67ee9c5662e-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-jw6px\" (UID: \"0bcc91a3-c35b-4498-8a55-c67ee9c5662e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" Sep 11 14:16:33.586791 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.586771 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bcc91a3-c35b-4498-8a55-c67ee9c5662e-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-jw6px\" (UID: \"0bcc91a3-c35b-4498-8a55-c67ee9c5662e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" Sep 11 14:16:33.586919 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.586872 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0bcc91a3-c35b-4498-8a55-c67ee9c5662e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-jw6px\" (UID: \"0bcc91a3-c35b-4498-8a55-c67ee9c5662e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" Sep 11 14:16:33.593954 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.593933 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r62zn\" (UniqueName: \"kubernetes.io/projected/0bcc91a3-c35b-4498-8a55-c67ee9c5662e-kube-api-access-r62zn\") pod \"prometheus-operator-68bcd7f987-jw6px\" (UID: \"0bcc91a3-c35b-4498-8a55-c67ee9c5662e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" Sep 11 14:16:33.689022 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:33.688956 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" Sep 11 14:16:35.806626 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:35.806411 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-jw6px"] Sep 11 14:16:35.808856 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:16:35.808801 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcc91a3_c35b_4498_8a55_c67ee9c5662e.slice/crio-eaaa39d42f8e038136630e3b34014d6100edee030011cef10bab40291014e9b5 WatchSource:0}: Error finding container eaaa39d42f8e038136630e3b34014d6100edee030011cef10bab40291014e9b5: Status 404 returned error can't find the container with id eaaa39d42f8e038136630e3b34014d6100edee030011cef10bab40291014e9b5 Sep 11 14:16:36.565176 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:36.565135 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj" event={"ID":"3aa047f7-a959-4f95-a04d-bdd9a415c65d","Type":"ContainerStarted","Data":"8a3243cb93e2fdf5e585b0f82eac8b4b9cb95aaeedb5cdaac405b6766c46220e"} Sep 11 14:16:36.565451 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:36.565359 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj" Sep 11 14:16:36.566789 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:36.566733 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" event={"ID":"0bcc91a3-c35b-4498-8a55-c67ee9c5662e","Type":"ContainerStarted","Data":"eaaa39d42f8e038136630e3b34014d6100edee030011cef10bab40291014e9b5"} Sep 11 14:16:36.567322 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:36.567301 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj" Sep 11 14:16:36.568365 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:36.568347 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-krm4c" event={"ID":"8bd8da52-9957-4092-b43d-d3f75992e3b5","Type":"ContainerStarted","Data":"700d52bb93352303fd5368e9c674a6b526ad8c7da60282452f991ed26dd2879c"} Sep 11 14:16:36.595593 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:36.595548 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-dc456b57c-k7jtj" podStartSLOduration=1.9906221290000001 podStartE2EDuration="6.595530298s" totalImagesPullingTime="4.604908169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:36.594068409 +0000 UTC m=+108.931036628" watchObservedRunningTime="2026-09-11 14:16:36.595530298 +0000 UTC m=+108.932498490" Sep 11 14:16:36.675243 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:36.675194 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-krm4c" podStartSLOduration=2.18415786 podStartE2EDuration="6.675178151s" totalImagesPullingTime="4.491020291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:36.673570201 +0000 UTC m=+109.010538393" watchObservedRunningTime="2026-09-11 14:16:36.675178151 +0000 UTC m=+109.012146342" Sep 11 14:16:37.573894 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:37.573859 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" event={"ID":"0bcc91a3-c35b-4498-8a55-c67ee9c5662e","Type":"ContainerStarted","Data":"62ee26bfa0b6dd2afacff399bc3a9408a114be571dca37294c4ae6ec2718c01f"} Sep 11 14:16:37.574343 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:37.573910 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" event={"ID":"0bcc91a3-c35b-4498-8a55-c67ee9c5662e","Type":"ContainerStarted","Data":"57643c7c8bc76fc1d597877a5f1fc7a6e8b355bbd8614aad7ab7932dd2f0e5a6"} Sep 11 14:16:37.597420 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:37.597376 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jw6px" podStartSLOduration=3.450969913 podStartE2EDuration="4.5973618s" totalImagesPullingTime="1.146391887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:37.59419765 +0000 UTC m=+109.931165842" watchObservedRunningTime="2026-09-11 14:16:37.5973618 +0000 UTC m=+109.934329990" Sep 11 14:16:39.543030 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:39.542996 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4gpp4" Sep 11 14:16:40.132853 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.132797 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq"] Sep 11 14:16:40.136100 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.136081 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.139722 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.139701 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 11 14:16:40.139934 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.139913 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 11 14:16:40.142189 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.142169 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 11 14:16:40.144866 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.144824 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5fbnl\"" Sep 11 14:16:40.154567 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.154542 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-67kmg"] Sep 11 14:16:40.157444 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.157427 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.159977 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.159927 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 14:16:40.160170 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.160152 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 14:16:40.162164 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.162145 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9rrqh\"" Sep 11 14:16:40.162454 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.162439 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 14:16:40.176503 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.176484 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq"] Sep 11 14:16:40.240446 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.240422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/89e0aaa5-26fa-424e-8407-bd04226f9172-node-exporter-wtmp\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.240561 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.240469 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/89e0aaa5-26fa-424e-8407-bd04226f9172-node-exporter-accelerators-collector-config\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.240561 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.240502 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/89e0aaa5-26fa-424e-8407-bd04226f9172-node-exporter-tls\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.240561 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.240533 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptth4\" (UniqueName: \"kubernetes.io/projected/89e0aaa5-26fa-424e-8407-bd04226f9172-kube-api-access-ptth4\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.240701 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.240566 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6208183e-4c87-4dd4-85e5-e436d602f0dd-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.240701 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.240597 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6208183e-4c87-4dd4-85e5-e436d602f0dd-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.240701 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.240623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6208183e-4c87-4dd4-85e5-e436d602f0dd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.240701 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.240652 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89e0aaa5-26fa-424e-8407-bd04226f9172-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.240854 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.240747 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drksz\" (UniqueName: \"kubernetes.io/projected/6208183e-4c87-4dd4-85e5-e436d602f0dd-kube-api-access-drksz\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.240854 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.240779 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/89e0aaa5-26fa-424e-8407-bd04226f9172-node-exporter-textfile\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.240854 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.240804 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89e0aaa5-26fa-424e-8407-bd04226f9172-metrics-client-ca\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.240854 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.240849 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6208183e-4c87-4dd4-85e5-e436d602f0dd-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.240998 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.240880 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6208183e-4c87-4dd4-85e5-e436d602f0dd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.240998 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.240904 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89e0aaa5-26fa-424e-8407-bd04226f9172-sys\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.240998 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.240930 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/89e0aaa5-26fa-424e-8407-bd04226f9172-root\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.341991 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.341967 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6208183e-4c87-4dd4-85e5-e436d602f0dd-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.342117 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.341999 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6208183e-4c87-4dd4-85e5-e436d602f0dd-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.342117 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342019 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6208183e-4c87-4dd4-85e5-e436d602f0dd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.342117 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89e0aaa5-26fa-424e-8407-bd04226f9172-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.342268 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342130 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drksz\" (UniqueName: \"kubernetes.io/projected/6208183e-4c87-4dd4-85e5-e436d602f0dd-kube-api-access-drksz\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.342268 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342160 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/89e0aaa5-26fa-424e-8407-bd04226f9172-node-exporter-textfile\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.342268 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342186 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89e0aaa5-26fa-424e-8407-bd04226f9172-metrics-client-ca\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.342268 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6208183e-4c87-4dd4-85e5-e436d602f0dd-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.342268 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342236 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6208183e-4c87-4dd4-85e5-e436d602f0dd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.342268 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342264 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89e0aaa5-26fa-424e-8407-bd04226f9172-sys\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.342559 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342294 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/89e0aaa5-26fa-424e-8407-bd04226f9172-root\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.342559 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342331 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/89e0aaa5-26fa-424e-8407-bd04226f9172-node-exporter-wtmp\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.342559 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342368 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/89e0aaa5-26fa-424e-8407-bd04226f9172-node-exporter-accelerators-collector-config\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.342559 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342396 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/89e0aaa5-26fa-424e-8407-bd04226f9172-node-exporter-tls\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.342559 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342425 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptth4\" (UniqueName: \"kubernetes.io/projected/89e0aaa5-26fa-424e-8407-bd04226f9172-kube-api-access-ptth4\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.342559 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342518 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6208183e-4c87-4dd4-85e5-e436d602f0dd-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.342559 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342548 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/89e0aaa5-26fa-424e-8407-bd04226f9172-root\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.342915 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342753 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6208183e-4c87-4dd4-85e5-e436d602f0dd-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.342915 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342768 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/89e0aaa5-26fa-424e-8407-bd04226f9172-node-exporter-textfile\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.342915 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342754 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/89e0aaa5-26fa-424e-8407-bd04226f9172-node-exporter-wtmp\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.342915 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342802 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89e0aaa5-26fa-424e-8407-bd04226f9172-metrics-client-ca\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.342915 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.342805 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89e0aaa5-26fa-424e-8407-bd04226f9172-sys\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.343225 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.343200 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/89e0aaa5-26fa-424e-8407-bd04226f9172-node-exporter-accelerators-collector-config\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.343346 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.343312 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6208183e-4c87-4dd4-85e5-e436d602f0dd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.344766 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.344736 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6208183e-4c87-4dd4-85e5-e436d602f0dd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.344961 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.344930 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/89e0aaa5-26fa-424e-8407-bd04226f9172-node-exporter-tls\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.345170 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.345151 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89e0aaa5-26fa-424e-8407-bd04226f9172-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.345329 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.345309 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6208183e-4c87-4dd4-85e5-e436d602f0dd-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.351400 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.351373 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drksz\" (UniqueName: \"kubernetes.io/projected/6208183e-4c87-4dd4-85e5-e436d602f0dd-kube-api-access-drksz\") pod \"kube-state-metrics-59fbdd77d-l9ctq\" (UID: \"6208183e-4c87-4dd4-85e5-e436d602f0dd\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.354045 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.354023 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptth4\" (UniqueName: \"kubernetes.io/projected/89e0aaa5-26fa-424e-8407-bd04226f9172-kube-api-access-ptth4\") pod \"node-exporter-67kmg\" (UID: \"89e0aaa5-26fa-424e-8407-bd04226f9172\") " pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.447748 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.447682 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" Sep 11 14:16:40.466314 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.466295 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-67kmg" Sep 11 14:16:40.477495 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:16:40.477471 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e0aaa5_26fa_424e_8407_bd04226f9172.slice/crio-54a952e18c3105353bafeaddb63ab5dec93678aafcdb71b6e9db94f95b51af37 WatchSource:0}: Error finding container 54a952e18c3105353bafeaddb63ab5dec93678aafcdb71b6e9db94f95b51af37: Status 404 returned error can't find the container with id 54a952e18c3105353bafeaddb63ab5dec93678aafcdb71b6e9db94f95b51af37 Sep 11 14:16:40.585606 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.585575 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-67kmg" event={"ID":"89e0aaa5-26fa-424e-8407-bd04226f9172","Type":"ContainerStarted","Data":"54a952e18c3105353bafeaddb63ab5dec93678aafcdb71b6e9db94f95b51af37"} Sep 11 14:16:40.729552 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:40.729526 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq"] Sep 11 14:16:40.731494 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:16:40.731465 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6208183e_4c87_4dd4_85e5_e436d602f0dd.slice/crio-835a425eda3365049b73356821967775ef6302ab38fb4a71c10ded6bc89aad26 WatchSource:0}: Error finding container 835a425eda3365049b73356821967775ef6302ab38fb4a71c10ded6bc89aad26: Status 404 returned error can't find the container with id 835a425eda3365049b73356821967775ef6302ab38fb4a71c10ded6bc89aad26 Sep 11 14:16:41.054526 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.054438 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:16:41.058250 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.058230 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.063302 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.063270 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 14:16:41.063302 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.063288 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 14:16:41.063581 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.063547 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 14:16:41.063581 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.063564 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 14:16:41.063754 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.063606 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 14:16:41.063872 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.063783 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 14:16:41.063937 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.063898 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8l5pn\"" Sep 11 14:16:41.063993 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.063977 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 14:16:41.064087 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.064071 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 14:16:41.067702 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.067683 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 14:16:41.083380 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.083358 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:16:41.148901 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.148801 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cp7j\" (UniqueName: \"kubernetes.io/projected/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-kube-api-access-2cp7j\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.148901 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.148858 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.148901 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.148888 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.149185 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.148948 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.149185 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.148988 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.149185 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.149018 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-config-volume\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.149185 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.149105 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-config-out\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.149185 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.149138 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.149185 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.149175 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.149517 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.149197 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.149517 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.149241 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-web-config\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.149517 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.149295 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.149517 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.149324 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.250707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.250572 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.250707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.250637 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.250707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.250666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-config-volume\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.250707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.250716 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-config-out\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.251082 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.250740 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.251082 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.250768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.251082 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.250790 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.251082 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.251054 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.251301 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:41.251281 2573 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 11 14:16:41.251369 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:16:41.251357 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-main-tls podName:d0237a6f-fc53-4839-8cb1-8803c6ee41a8 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:41.751336 +0000 UTC m=+114.088304184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "d0237a6f-fc53-4839-8cb1-8803c6ee41a8") : secret "alertmanager-main-tls" not found Sep 11 14:16:41.251765 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.251479 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-web-config\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.251765 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.251529 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.251765 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.251552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.251765 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.251582 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cp7j\" (UniqueName: \"kubernetes.io/projected/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-kube-api-access-2cp7j\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.251765 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.251610 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.251765 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.251639 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.255123 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.254760 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.255123 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.254769 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.255283 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.255213 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.255691 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.255673 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-web-config\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.257368 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.257342 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.257660 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.257612 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.257660 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.257638 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.258024 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.258001 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-config-out\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.258314 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.258277 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-config-volume\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.259568 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.259525 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.271204 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.271164 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cp7j\" (UniqueName: \"kubernetes.io/projected/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-kube-api-access-2cp7j\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.591527 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.591500 2573 generic.go:356] "Generic (PLEG): container finished" podID="89e0aaa5-26fa-424e-8407-bd04226f9172" containerID="36b15c73376daae22cfe55fc69acbc3fa22954c02d18e7610563eb9dd66e4c34" exitCode=0 Sep 11 14:16:41.591902 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.591598 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-67kmg" event={"ID":"89e0aaa5-26fa-424e-8407-bd04226f9172","Type":"ContainerDied","Data":"36b15c73376daae22cfe55fc69acbc3fa22954c02d18e7610563eb9dd66e4c34"} Sep 11 14:16:41.592801 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.592765 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" event={"ID":"6208183e-4c87-4dd4-85e5-e436d602f0dd","Type":"ContainerStarted","Data":"835a425eda3365049b73356821967775ef6302ab38fb4a71c10ded6bc89aad26"} Sep 11 14:16:41.757889 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.757786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.761350 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.761313 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:41.970818 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:41.970788 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:42.244527 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.244493 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-786c97fb9c-p22rl"] Sep 11 14:16:42.248079 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.248062 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.263506 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.263452 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 11 14:16:42.263723 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.263706 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bmo1l18u9kkbl\"" Sep 11 14:16:42.264769 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.264754 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 11 14:16:42.265009 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.264993 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 11 14:16:42.265923 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.265908 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 11 14:16:42.267074 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.267057 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 11 14:16:42.299126 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.299109 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-h6q7l\"" Sep 11 14:16:42.341706 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:16:42.341682 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0237a6f_fc53_4839_8cb1_8803c6ee41a8.slice/crio-87a373456b79be8d8e3aee34677bea90ac7f061825962c2da8b43e7c8fbc763d WatchSource:0}: Error finding container 87a373456b79be8d8e3aee34677bea90ac7f061825962c2da8b43e7c8fbc763d: Status 404 returned error can't find the container with id 87a373456b79be8d8e3aee34677bea90ac7f061825962c2da8b43e7c8fbc763d Sep 11 14:16:42.348180 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.348156 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:16:42.363792 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.363771 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.363900 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.363813 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b15ffddc-f04a-4ee0-b396-e9a4e3883808-metrics-client-ca\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.363900 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.363867 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.363988 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.363923 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.363988 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.363962 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.364070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.363989 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxdsj\" (UniqueName: \"kubernetes.io/projected/b15ffddc-f04a-4ee0-b396-e9a4e3883808-kube-api-access-zxdsj\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.364070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.364029 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-grpc-tls\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.364070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.364047 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-thanos-querier-tls\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.397177 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.397153 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-786c97fb9c-p22rl"] Sep 11 14:16:42.464637 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.464606 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.464782 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.464647 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.464782 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.464669 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxdsj\" (UniqueName: \"kubernetes.io/projected/b15ffddc-f04a-4ee0-b396-e9a4e3883808-kube-api-access-zxdsj\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.464782 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.464707 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-grpc-tls\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.464782 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.464728 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-thanos-querier-tls\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.464782 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.464751 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.465039 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.464791 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b15ffddc-f04a-4ee0-b396-e9a4e3883808-metrics-client-ca\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.465039 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.464821 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.465742 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.465715 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b15ffddc-f04a-4ee0-b396-e9a4e3883808-metrics-client-ca\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.467892 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.467825 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-thanos-querier-tls\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.467997 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.467930 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.467997 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.467947 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.467997 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.467934 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-grpc-tls\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.468134 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.468067 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.468425 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.468408 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b15ffddc-f04a-4ee0-b396-e9a4e3883808-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.485151 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.485129 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxdsj\" (UniqueName: \"kubernetes.io/projected/b15ffddc-f04a-4ee0-b396-e9a4e3883808-kube-api-access-zxdsj\") pod \"thanos-querier-786c97fb9c-p22rl\" (UID: \"b15ffddc-f04a-4ee0-b396-e9a4e3883808\") " pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.556697 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.556632 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:42.599846 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.599797 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-67kmg" event={"ID":"89e0aaa5-26fa-424e-8407-bd04226f9172","Type":"ContainerStarted","Data":"c9dee2258e952b41a609717e262e45bd3a87660871345f42c5f4901bfb3e4feb"} Sep 11 14:16:42.600243 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.599915 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-67kmg" event={"ID":"89e0aaa5-26fa-424e-8407-bd04226f9172","Type":"ContainerStarted","Data":"c586e464fa0a2acc4d37fe528c369da9e53e52df05112486943c2d9277150341"} Sep 11 14:16:42.601168 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.601145 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0237a6f-fc53-4839-8cb1-8803c6ee41a8","Type":"ContainerStarted","Data":"87a373456b79be8d8e3aee34677bea90ac7f061825962c2da8b43e7c8fbc763d"} Sep 11 14:16:42.603407 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.603381 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" event={"ID":"6208183e-4c87-4dd4-85e5-e436d602f0dd","Type":"ContainerStarted","Data":"25dd9851080a90b41602d85561cf3f3724006ff0746c108779c8f1641cce42aa"} Sep 11 14:16:42.603536 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.603432 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" event={"ID":"6208183e-4c87-4dd4-85e5-e436d602f0dd","Type":"ContainerStarted","Data":"6c70ea268317f59df5c98c2adcfbe4a8cad2901d221221b69cb4be9250f64dda"} Sep 11 14:16:42.603536 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.603473 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" event={"ID":"6208183e-4c87-4dd4-85e5-e436d602f0dd","Type":"ContainerStarted","Data":"35a6668b9e2d6604fa4f11c07750d4bc49a256600a5484ac164a6a526b9562cc"} Sep 11 14:16:42.642076 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.642033 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-67kmg" podStartSLOduration=1.64577267 podStartE2EDuration="2.642018148s" totalImagesPullingTime="935.342551ms" totalInitContainerRuntime="60.902927ms" isStatefulPod=true podCreationTimestamp="2026-09-11 14:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:42.640071463 +0000 UTC m=+114.977039653" watchObservedRunningTime="2026-09-11 14:16:42.642018148 +0000 UTC m=+114.978986333" Sep 11 14:16:42.683486 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.683447 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l9ctq" podStartSLOduration=1.404605497 podStartE2EDuration="2.683426786s" totalImagesPullingTime="1.278821289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:42.682701611 +0000 UTC m=+115.019669801" watchObservedRunningTime="2026-09-11 14:16:42.683426786 +0000 UTC m=+115.020394977" Sep 11 14:16:42.864020 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:16:42.863978 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15ffddc_f04a_4ee0_b396_e9a4e3883808.slice/crio-9377e2c76e75a2fe7a35259df3310a3d73fd6dc153a3465d348491adfc3c7fe5 WatchSource:0}: Error finding container 9377e2c76e75a2fe7a35259df3310a3d73fd6dc153a3465d348491adfc3c7fe5: Status 404 returned error can't find the container with id 9377e2c76e75a2fe7a35259df3310a3d73fd6dc153a3465d348491adfc3c7fe5 Sep 11 14:16:42.868934 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:42.868902 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-786c97fb9c-p22rl"] Sep 11 14:16:43.609916 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:43.609881 2573 generic.go:356] "Generic (PLEG): container finished" podID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerID="8e7f3eadca4abc87861388070926bad750e5144007c1d236ed9c75fe8a9fd351" exitCode=0 Sep 11 14:16:43.610351 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:43.609981 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0237a6f-fc53-4839-8cb1-8803c6ee41a8","Type":"ContainerDied","Data":"8e7f3eadca4abc87861388070926bad750e5144007c1d236ed9c75fe8a9fd351"} Sep 11 14:16:43.612234 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:43.612203 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" event={"ID":"b15ffddc-f04a-4ee0-b396-e9a4e3883808","Type":"ContainerStarted","Data":"9377e2c76e75a2fe7a35259df3310a3d73fd6dc153a3465d348491adfc3c7fe5"} Sep 11 14:16:45.058927 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.058892 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-546c45558c-vlls4"] Sep 11 14:16:45.062160 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.062140 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.068028 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.068005 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2trbcocbultdd\"" Sep 11 14:16:45.069202 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.069169 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 11 14:16:45.089262 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.088007 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 11 14:16:45.089262 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.088221 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 14:16:45.089262 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.088667 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4gvvd\"" Sep 11 14:16:45.089262 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.088903 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 11 14:16:45.103594 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.103568 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-546c45558c-vlls4"] Sep 11 14:16:45.191726 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.191693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/015081a7-847f-458f-a5de-eb1662c35b6f-metrics-server-audit-profiles\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.191881 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.191740 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/015081a7-847f-458f-a5de-eb1662c35b6f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.191881 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.191772 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/015081a7-847f-458f-a5de-eb1662c35b6f-client-ca-bundle\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.191881 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.191819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/015081a7-847f-458f-a5de-eb1662c35b6f-secret-metrics-server-client-certs\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.191986 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.191876 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/015081a7-847f-458f-a5de-eb1662c35b6f-secret-metrics-server-tls\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.191986 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.191930 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/015081a7-847f-458f-a5de-eb1662c35b6f-audit-log\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.192044 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.192010 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2m4b\" (UniqueName: \"kubernetes.io/projected/015081a7-847f-458f-a5de-eb1662c35b6f-kube-api-access-j2m4b\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.293233 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.293209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/015081a7-847f-458f-a5de-eb1662c35b6f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.293323 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.293252 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/015081a7-847f-458f-a5de-eb1662c35b6f-client-ca-bundle\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.293323 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.293296 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/015081a7-847f-458f-a5de-eb1662c35b6f-secret-metrics-server-client-certs\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.293323 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.293316 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/015081a7-847f-458f-a5de-eb1662c35b6f-secret-metrics-server-tls\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.293466 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.293353 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/015081a7-847f-458f-a5de-eb1662c35b6f-audit-log\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.293466 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.293392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2m4b\" (UniqueName: \"kubernetes.io/projected/015081a7-847f-458f-a5de-eb1662c35b6f-kube-api-access-j2m4b\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.293466 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.293424 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/015081a7-847f-458f-a5de-eb1662c35b6f-metrics-server-audit-profiles\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.294381 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.294053 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/015081a7-847f-458f-a5de-eb1662c35b6f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.294381 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.294332 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/015081a7-847f-458f-a5de-eb1662c35b6f-audit-log\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.294746 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.294727 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/015081a7-847f-458f-a5de-eb1662c35b6f-metrics-server-audit-profiles\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.297271 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.297138 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/015081a7-847f-458f-a5de-eb1662c35b6f-secret-metrics-server-client-certs\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.298032 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.298000 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/015081a7-847f-458f-a5de-eb1662c35b6f-secret-metrics-server-tls\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.298417 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.298398 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/015081a7-847f-458f-a5de-eb1662c35b6f-client-ca-bundle\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.468095 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.468054 2573 patch_prober.go:28] interesting pod/image-registry-856c4b595b-4k97s container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 11 14:16:45.468196 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.468109 2573 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-856c4b595b-4k97s" podUID="32273522-7dd5-42c8-b2ce-fb3c46906000" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:16:45.545016 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.544995 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2m4b\" (UniqueName: \"kubernetes.io/projected/015081a7-847f-458f-a5de-eb1662c35b6f-kube-api-access-j2m4b\") pod \"metrics-server-546c45558c-vlls4\" (UID: \"015081a7-847f-458f-a5de-eb1662c35b6f\") " pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:45.624999 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.624962 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0237a6f-fc53-4839-8cb1-8803c6ee41a8","Type":"ContainerStarted","Data":"59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4"} Sep 11 14:16:45.625137 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.625045 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0237a6f-fc53-4839-8cb1-8803c6ee41a8","Type":"ContainerStarted","Data":"830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192"} Sep 11 14:16:45.625137 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.625093 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0237a6f-fc53-4839-8cb1-8803c6ee41a8","Type":"ContainerStarted","Data":"786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9"} Sep 11 14:16:45.625137 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.625136 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0237a6f-fc53-4839-8cb1-8803c6ee41a8","Type":"ContainerStarted","Data":"f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1"} Sep 11 14:16:45.625292 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.625177 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0237a6f-fc53-4839-8cb1-8803c6ee41a8","Type":"ContainerStarted","Data":"07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82"} Sep 11 14:16:45.627602 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.627581 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" event={"ID":"b15ffddc-f04a-4ee0-b396-e9a4e3883808","Type":"ContainerStarted","Data":"6da4e68ab1f0788a4ea4e1e85c3df29f35c3ecee85b4110e67254756629b090d"} Sep 11 14:16:45.627704 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.627639 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" event={"ID":"b15ffddc-f04a-4ee0-b396-e9a4e3883808","Type":"ContainerStarted","Data":"33e627d6521860af1d482c6c10ebdd52af83f9210abd41d4a36bac488aa1c80e"} Sep 11 14:16:45.627704 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.627668 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" event={"ID":"b15ffddc-f04a-4ee0-b396-e9a4e3883808","Type":"ContainerStarted","Data":"9c1263ac3b7b8050ac256ca3d73052d1e34a17df0d014f6c2c28fc32670082fc"} Sep 11 14:16:45.673644 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:45.673618 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:16:46.143911 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.143874 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-546c45558c-vlls4"] Sep 11 14:16:46.227103 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.227065 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6c88d674d4-h52n8"] Sep 11 14:16:46.232757 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.232728 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.237867 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.237495 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 11 14:16:46.237867 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.237663 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 11 14:16:46.238049 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.237947 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7fbx9\"" Sep 11 14:16:46.238437 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.238209 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 11 14:16:46.238437 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.238274 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 11 14:16:46.238437 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.238333 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 11 14:16:46.249280 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.249260 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 11 14:16:46.256535 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:16:46.256514 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015081a7_847f_458f_a5de_eb1662c35b6f.slice/crio-a229b204d57ae61005a84609fec96b4b4193cd04d4b6de10d43d64fae62082cf WatchSource:0}: Error finding container a229b204d57ae61005a84609fec96b4b4193cd04d4b6de10d43d64fae62082cf: Status 404 returned error can't find the container with id a229b204d57ae61005a84609fec96b4b4193cd04d4b6de10d43d64fae62082cf Sep 11 14:16:46.286248 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.286222 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c88d674d4-h52n8"] Sep 11 14:16:46.301508 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.301485 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8sk\" (UniqueName: \"kubernetes.io/projected/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-kube-api-access-nj8sk\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.301599 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.301515 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-secret-telemeter-client\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.301599 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.301544 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-serving-certs-ca-bundle\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.301701 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.301588 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-telemeter-client-tls\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.301701 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.301662 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.301795 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.301729 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-federate-client-tls\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.301856 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.301795 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.301856 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.301823 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-metrics-client-ca\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.403161 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.403089 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.403161 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.403133 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-federate-client-tls\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.403305 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.403209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.403305 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.403236 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-metrics-client-ca\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.403305 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.403258 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8sk\" (UniqueName: \"kubernetes.io/projected/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-kube-api-access-nj8sk\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.403305 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.403283 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-secret-telemeter-client\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.403496 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.403315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-serving-certs-ca-bundle\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.403496 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.403336 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-telemeter-client-tls\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.404132 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.404092 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.404132 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.404118 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-metrics-client-ca\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.404330 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.404145 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-serving-certs-ca-bundle\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.405779 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.405750 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.405904 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.405883 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-federate-client-tls\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.406123 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.406104 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-telemeter-client-tls\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.406200 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.406174 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-secret-telemeter-client\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.426373 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.426353 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8sk\" (UniqueName: \"kubernetes.io/projected/ce41dbc1-7d2f-47a6-b809-55eb41019d6c-kube-api-access-nj8sk\") pod \"telemeter-client-6c88d674d4-h52n8\" (UID: \"ce41dbc1-7d2f-47a6-b809-55eb41019d6c\") " pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.553749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.553713 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" Sep 11 14:16:46.633008 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.632977 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-546c45558c-vlls4" event={"ID":"015081a7-847f-458f-a5de-eb1662c35b6f","Type":"ContainerStarted","Data":"a229b204d57ae61005a84609fec96b4b4193cd04d4b6de10d43d64fae62082cf"} Sep 11 14:16:46.636199 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.636172 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0237a6f-fc53-4839-8cb1-8803c6ee41a8","Type":"ContainerStarted","Data":"5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311"} Sep 11 14:16:46.881357 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.881294 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.538217531 podStartE2EDuration="5.881274118s" totalImagesPullingTime="3.32971208s" totalInitContainerRuntime="13.344507ms" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:46.774701012 +0000 UTC m=+119.111669213" watchObservedRunningTime="2026-09-11 14:16:46.881274118 +0000 UTC m=+119.218242310" Sep 11 14:16:46.881714 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:46.881694 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c88d674d4-h52n8"] Sep 11 14:16:46.885101 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:16:46.885071 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce41dbc1_7d2f_47a6_b809_55eb41019d6c.slice/crio-0dae62cfa46af48f8a886f4da1a7e1a3d7888843d042fb6a19561aa9d476f915 WatchSource:0}: Error finding container 0dae62cfa46af48f8a886f4da1a7e1a3d7888843d042fb6a19561aa9d476f915: Status 404 returned error can't find the container with id 0dae62cfa46af48f8a886f4da1a7e1a3d7888843d042fb6a19561aa9d476f915 Sep 11 14:16:47.648508 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:47.647963 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" event={"ID":"b15ffddc-f04a-4ee0-b396-e9a4e3883808","Type":"ContainerStarted","Data":"1ac401b002f02b01e0d5c567df9e6d8d6e4b42eafeb023abc5879b29a4453f16"} Sep 11 14:16:47.648508 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:47.648060 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" event={"ID":"b15ffddc-f04a-4ee0-b396-e9a4e3883808","Type":"ContainerStarted","Data":"cdef634e52848e1e107075f911eaf986c1161733eaec0628b09169d02a24906e"} Sep 11 14:16:47.648508 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:47.648106 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" event={"ID":"b15ffddc-f04a-4ee0-b396-e9a4e3883808","Type":"ContainerStarted","Data":"e737fc3b5b8719699ecbadcef874bffa8e431ac552495d13b66489f964e877a8"} Sep 11 14:16:47.648508 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:47.648267 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:47.650882 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:47.650806 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" event={"ID":"ce41dbc1-7d2f-47a6-b809-55eb41019d6c","Type":"ContainerStarted","Data":"0dae62cfa46af48f8a886f4da1a7e1a3d7888843d042fb6a19561aa9d476f915"} Sep 11 14:16:48.540892 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:48.540857 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:16:48.632237 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:48.632148 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" podStartSLOduration=3.046033637 podStartE2EDuration="6.632128662s" totalImagesPullingTime="3.586095025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:47.779672589 +0000 UTC m=+120.116640782" watchObservedRunningTime="2026-09-11 14:16:48.632128662 +0000 UTC m=+120.969096854" Sep 11 14:16:48.655899 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:48.655867 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-546c45558c-vlls4" event={"ID":"015081a7-847f-458f-a5de-eb1662c35b6f","Type":"ContainerStarted","Data":"a0d3e2bbea0772a7134a3a7458cb242186b7166ebe67ebf8878a2d22171f5127"} Sep 11 14:16:49.662351 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:49.662313 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" event={"ID":"ce41dbc1-7d2f-47a6-b809-55eb41019d6c","Type":"ContainerStarted","Data":"1f8e8d70206fb58d0dcff15b19766937912703d85f37789808895f9b621c52e6"} Sep 11 14:16:49.662748 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:49.662370 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" event={"ID":"ce41dbc1-7d2f-47a6-b809-55eb41019d6c","Type":"ContainerStarted","Data":"fae87458e1165d6e03e0241bdfc59a2804f1038f4718ce4712e29dffc1ef5339"} Sep 11 14:16:49.662748 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:49.662388 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" event={"ID":"ce41dbc1-7d2f-47a6-b809-55eb41019d6c","Type":"ContainerStarted","Data":"bfa6ddef7a8e00c0d0289667b647f1fca48412bef371805dbf18554f43e8b7ab"} Sep 11 14:16:49.747019 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:49.746966 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-546c45558c-vlls4" podStartSLOduration=4.212625169 podStartE2EDuration="5.746949719s" totalImagesPullingTime="1.53432455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:48.795586144 +0000 UTC m=+121.132554334" watchObservedRunningTime="2026-09-11 14:16:49.746949719 +0000 UTC m=+122.083917909" Sep 11 14:16:49.748027 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:49.747986 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6c88d674d4-h52n8" podStartSLOduration=2.02082337 podStartE2EDuration="3.747969699s" totalImagesPullingTime="1.727146329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:49.744642833 +0000 UTC m=+122.081611015" watchObservedRunningTime="2026-09-11 14:16:49.747969699 +0000 UTC m=+122.084937914" Sep 11 14:16:53.177359 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:53.177329 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-856c4b595b-4k97s"] Sep 11 14:16:53.663724 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:53.663696 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-786c97fb9c-p22rl" Sep 11 14:16:57.906733 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:57.906690 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs\") pod \"network-metrics-daemon-fh5nz\" (UID: \"afa6cfac-b37f-4e64-a308-658d8a3b62d9\") " pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:16:57.909130 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:57.909107 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa6cfac-b37f-4e64-a308-658d8a3b62d9-metrics-certs\") pod \"network-metrics-daemon-fh5nz\" (UID: \"afa6cfac-b37f-4e64-a308-658d8a3b62d9\") " pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:16:57.998030 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:57.998000 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vtjb2\"" Sep 11 14:16:58.002994 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:58.002976 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh5nz" Sep 11 14:16:58.299656 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:58.299579 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fh5nz"] Sep 11 14:16:58.302726 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:16:58.302698 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa6cfac_b37f_4e64_a308_658d8a3b62d9.slice/crio-af6d3fe13675321474c4daf9e1c5179fb32b51650da24798b76619040ff30982 WatchSource:0}: Error finding container af6d3fe13675321474c4daf9e1c5179fb32b51650da24798b76619040ff30982: Status 404 returned error can't find the container with id af6d3fe13675321474c4daf9e1c5179fb32b51650da24798b76619040ff30982 Sep 11 14:16:58.699407 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:58.699373 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fh5nz" event={"ID":"afa6cfac-b37f-4e64-a308-658d8a3b62d9","Type":"ContainerStarted","Data":"af6d3fe13675321474c4daf9e1c5179fb32b51650da24798b76619040ff30982"} Sep 11 14:16:59.705200 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:59.705162 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fh5nz" event={"ID":"afa6cfac-b37f-4e64-a308-658d8a3b62d9","Type":"ContainerStarted","Data":"5b844f0a8e6721650f007405c6edf7416639d222da7a6523c7859e48472e8b5a"} Sep 11 14:16:59.705561 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:59.705217 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fh5nz" event={"ID":"afa6cfac-b37f-4e64-a308-658d8a3b62d9","Type":"ContainerStarted","Data":"ad31171e7b0f811909410b0786be407665f58c783b96400098c59850ac2c01a1"} Sep 11 14:16:59.763922 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:16:59.763874 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fh5nz" podStartSLOduration=130.802495219 podStartE2EDuration="2m11.763858526s" totalImagesPullingTime="961.363307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:59.762857768 +0000 UTC m=+132.099825956" watchObservedRunningTime="2026-09-11 14:16:59.763858526 +0000 UTC m=+132.100826722" Sep 11 14:17:00.853269 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:00.853236 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-6qbfq"] Sep 11 14:17:00.856557 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:00.856541 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-6qbfq" Sep 11 14:17:00.868979 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:00.868960 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-8btj8\"" Sep 11 14:17:00.869241 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:00.869227 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 14:17:00.869277 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:00.869239 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 14:17:00.932453 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:00.932426 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcb8w\" (UniqueName: \"kubernetes.io/projected/fcc91ee5-a37f-42dc-a11d-b719dcae387a-kube-api-access-xcb8w\") pod \"downloads-76f99584f4-6qbfq\" (UID: \"fcc91ee5-a37f-42dc-a11d-b719dcae387a\") " pod="openshift-console/downloads-76f99584f4-6qbfq" Sep 11 14:17:01.002380 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:01.002348 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-6qbfq"] Sep 11 14:17:01.033327 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:01.033299 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcb8w\" (UniqueName: \"kubernetes.io/projected/fcc91ee5-a37f-42dc-a11d-b719dcae387a-kube-api-access-xcb8w\") pod \"downloads-76f99584f4-6qbfq\" (UID: \"fcc91ee5-a37f-42dc-a11d-b719dcae387a\") " pod="openshift-console/downloads-76f99584f4-6qbfq" Sep 11 14:17:01.059718 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:01.059690 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcb8w\" (UniqueName: \"kubernetes.io/projected/fcc91ee5-a37f-42dc-a11d-b719dcae387a-kube-api-access-xcb8w\") pod \"downloads-76f99584f4-6qbfq\" (UID: \"fcc91ee5-a37f-42dc-a11d-b719dcae387a\") " pod="openshift-console/downloads-76f99584f4-6qbfq" Sep 11 14:17:01.165491 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:01.165409 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-6qbfq" Sep 11 14:17:01.484881 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:01.484793 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-6qbfq"] Sep 11 14:17:01.487612 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:17:01.487568 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc91ee5_a37f_42dc_a11d_b719dcae387a.slice/crio-ad104cb647c44a63ff591497d407977ba96dc5c643eec409d432884dbc0cc25f WatchSource:0}: Error finding container ad104cb647c44a63ff591497d407977ba96dc5c643eec409d432884dbc0cc25f: Status 404 returned error can't find the container with id ad104cb647c44a63ff591497d407977ba96dc5c643eec409d432884dbc0cc25f Sep 11 14:17:01.716115 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:01.716078 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-6qbfq" event={"ID":"fcc91ee5-a37f-42dc-a11d-b719dcae387a","Type":"ContainerStarted","Data":"ad104cb647c44a63ff591497d407977ba96dc5c643eec409d432884dbc0cc25f"} Sep 11 14:17:05.674603 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:05.674558 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:17:05.675032 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:05.674638 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:17:17.792911 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:17.792816 2573 generic.go:356] "Generic (PLEG): container finished" podID="653a4ec0-e4b5-4956-9fe8-969d4eb6a093" containerID="c02fbbc0b820aa625e2c67e7fc4a368fb79d7b7deaf220f6c2147d82d43a3352" exitCode=0 Sep 11 14:17:17.792911 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:17.792865 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z" event={"ID":"653a4ec0-e4b5-4956-9fe8-969d4eb6a093","Type":"ContainerDied","Data":"c02fbbc0b820aa625e2c67e7fc4a368fb79d7b7deaf220f6c2147d82d43a3352"} Sep 11 14:17:17.793396 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:17.793229 2573 scope.go:117] "RemoveContainer" containerID="c02fbbc0b820aa625e2c67e7fc4a368fb79d7b7deaf220f6c2147d82d43a3352" Sep 11 14:17:17.794501 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:17.794472 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-6qbfq" event={"ID":"fcc91ee5-a37f-42dc-a11d-b719dcae387a","Type":"ContainerStarted","Data":"03c1ace82fc4dd41126e698f5835a457b39dcf92af1b6d54fb4e2416852e10d6"} Sep 11 14:17:17.794647 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:17.794630 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-6qbfq" Sep 11 14:17:17.795919 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:17.795888 2573 patch_prober.go:28] interesting pod/downloads-76f99584f4-6qbfq container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.20:8080/\": dial tcp 10.132.0.20:8080: connect: connection refused" start-of-body= Sep 11 14:17:17.795990 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:17.795931 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-6qbfq" podUID="fcc91ee5-a37f-42dc-a11d-b719dcae387a" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.20:8080/\": dial tcp 10.132.0.20:8080: connect: connection refused" Sep 11 14:17:17.869536 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:17.869488 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-6qbfq" podStartSLOduration=1.854562068 podStartE2EDuration="17.869470828s" totalImagesPullingTime="16.01490876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:17.86940301 +0000 UTC m=+150.206371201" watchObservedRunningTime="2026-09-11 14:17:17.869470828 +0000 UTC m=+150.206439019" Sep 11 14:17:18.196373 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.196334 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-856c4b595b-4k97s" podUID="32273522-7dd5-42c8-b2ce-fb3c46906000" containerName="registry" containerID="cri-o://a4b0665fd4e2df8a70b62645b517c603927b68e23fc03ffcd539a76aeb2470e1" gracePeriod=30 Sep 11 14:17:18.376186 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.375910 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b4d6f4ddb-mn4gk"] Sep 11 14:17:18.407773 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.398896 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.414780 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.413681 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 14:17:18.416436 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.416404 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b4d6f4ddb-mn4gk"] Sep 11 14:17:18.418520 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.418336 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 14:17:18.418795 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.418758 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 14:17:18.419236 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.419215 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4nz8j\"" Sep 11 14:17:18.419489 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.419469 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 14:17:18.419932 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.419769 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 14:17:18.428229 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.428208 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 14:17:18.501279 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.501246 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-oauth-serving-cert\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.501417 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.501306 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-trusted-ca-bundle\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.501417 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.501380 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-service-ca\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.501541 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.501442 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cd05741-42cc-45aa-a525-f0f24dce2105-console-serving-cert\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.501541 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.501477 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzf7b\" (UniqueName: \"kubernetes.io/projected/9cd05741-42cc-45aa-a525-f0f24dce2105-kube-api-access-rzf7b\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.501541 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.501514 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-console-config\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.501683 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.501545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cd05741-42cc-45aa-a525-f0f24dce2105-console-oauth-config\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.522151 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.522124 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:17:18.602153 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.602117 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-trusted-ca-bundle\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.602342 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.602168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-service-ca\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.602342 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.602225 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cd05741-42cc-45aa-a525-f0f24dce2105-console-serving-cert\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.602342 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.602255 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzf7b\" (UniqueName: \"kubernetes.io/projected/9cd05741-42cc-45aa-a525-f0f24dce2105-kube-api-access-rzf7b\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.602342 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.602288 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-console-config\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.602342 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.602315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cd05741-42cc-45aa-a525-f0f24dce2105-console-oauth-config\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.602615 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.602409 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-oauth-serving-cert\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.603103 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.603079 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-oauth-serving-cert\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.603103 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.603078 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-trusted-ca-bundle\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.603935 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.603640 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-service-ca\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.604134 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.604096 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-console-config\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.606180 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.606151 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cd05741-42cc-45aa-a525-f0f24dce2105-console-serving-cert\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.606279 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.606154 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cd05741-42cc-45aa-a525-f0f24dce2105-console-oauth-config\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.620659 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.620633 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzf7b\" (UniqueName: \"kubernetes.io/projected/9cd05741-42cc-45aa-a525-f0f24dce2105-kube-api-access-rzf7b\") pod \"console-b4d6f4ddb-mn4gk\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.703221 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.703178 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32273522-7dd5-42c8-b2ce-fb3c46906000-image-registry-private-configuration\") pod \"32273522-7dd5-42c8-b2ce-fb3c46906000\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " Sep 11 14:17:18.703426 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.703266 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32273522-7dd5-42c8-b2ce-fb3c46906000-trusted-ca\") pod \"32273522-7dd5-42c8-b2ce-fb3c46906000\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " Sep 11 14:17:18.703426 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.703297 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wknfs\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-kube-api-access-wknfs\") pod \"32273522-7dd5-42c8-b2ce-fb3c46906000\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " Sep 11 14:17:18.703426 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.703330 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32273522-7dd5-42c8-b2ce-fb3c46906000-ca-trust-extracted\") pod \"32273522-7dd5-42c8-b2ce-fb3c46906000\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " Sep 11 14:17:18.703426 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.703374 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-bound-sa-token\") pod \"32273522-7dd5-42c8-b2ce-fb3c46906000\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " Sep 11 14:17:18.703644 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.703439 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-certificates\") pod \"32273522-7dd5-42c8-b2ce-fb3c46906000\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " Sep 11 14:17:18.703644 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.703485 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32273522-7dd5-42c8-b2ce-fb3c46906000-installation-pull-secrets\") pod \"32273522-7dd5-42c8-b2ce-fb3c46906000\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " Sep 11 14:17:18.703644 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.703511 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls\") pod \"32273522-7dd5-42c8-b2ce-fb3c46906000\" (UID: \"32273522-7dd5-42c8-b2ce-fb3c46906000\") " Sep 11 14:17:18.703792 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.703666 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32273522-7dd5-42c8-b2ce-fb3c46906000-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "32273522-7dd5-42c8-b2ce-fb3c46906000" (UID: "32273522-7dd5-42c8-b2ce-fb3c46906000"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:17:18.703872 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.703818 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32273522-7dd5-42c8-b2ce-fb3c46906000-trusted-ca\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:17:18.704386 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.704132 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "32273522-7dd5-42c8-b2ce-fb3c46906000" (UID: "32273522-7dd5-42c8-b2ce-fb3c46906000"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:17:18.706393 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.706338 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "32273522-7dd5-42c8-b2ce-fb3c46906000" (UID: "32273522-7dd5-42c8-b2ce-fb3c46906000"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:17:18.706393 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.706351 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32273522-7dd5-42c8-b2ce-fb3c46906000-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "32273522-7dd5-42c8-b2ce-fb3c46906000" (UID: "32273522-7dd5-42c8-b2ce-fb3c46906000"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:17:18.706866 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.706812 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-kube-api-access-wknfs" (OuterVolumeSpecName: "kube-api-access-wknfs") pod "32273522-7dd5-42c8-b2ce-fb3c46906000" (UID: "32273522-7dd5-42c8-b2ce-fb3c46906000"). InnerVolumeSpecName "kube-api-access-wknfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:17:18.707298 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.707267 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "32273522-7dd5-42c8-b2ce-fb3c46906000" (UID: "32273522-7dd5-42c8-b2ce-fb3c46906000"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:17:18.707637 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.707612 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32273522-7dd5-42c8-b2ce-fb3c46906000-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "32273522-7dd5-42c8-b2ce-fb3c46906000" (UID: "32273522-7dd5-42c8-b2ce-fb3c46906000"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:17:18.714490 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.714453 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32273522-7dd5-42c8-b2ce-fb3c46906000-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "32273522-7dd5-42c8-b2ce-fb3c46906000" (UID: "32273522-7dd5-42c8-b2ce-fb3c46906000"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:17:18.718769 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.718743 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:18.802574 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.802541 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-4g66z" event={"ID":"653a4ec0-e4b5-4956-9fe8-969d4eb6a093","Type":"ContainerStarted","Data":"72bbfae691f2b3abfd9d74da3c6d4e82ad8c4a72932c2209fbe5072188768a00"} Sep 11 14:17:18.805493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.804122 2573 generic.go:356] "Generic (PLEG): container finished" podID="32273522-7dd5-42c8-b2ce-fb3c46906000" containerID="a4b0665fd4e2df8a70b62645b517c603927b68e23fc03ffcd539a76aeb2470e1" exitCode=0 Sep 11 14:17:18.805493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.804231 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-856c4b595b-4k97s" Sep 11 14:17:18.805493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.804245 2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32273522-7dd5-42c8-b2ce-fb3c46906000-installation-pull-secrets\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:17:18.805493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.804260 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-856c4b595b-4k97s" event={"ID":"32273522-7dd5-42c8-b2ce-fb3c46906000","Type":"ContainerDied","Data":"a4b0665fd4e2df8a70b62645b517c603927b68e23fc03ffcd539a76aeb2470e1"} Sep 11 14:17:18.805493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.804270 2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-tls\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:17:18.805493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.804286 2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32273522-7dd5-42c8-b2ce-fb3c46906000-image-registry-private-configuration\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:17:18.805493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.804301 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wknfs\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-kube-api-access-wknfs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:17:18.805493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.804310 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-856c4b595b-4k97s" event={"ID":"32273522-7dd5-42c8-b2ce-fb3c46906000","Type":"ContainerDied","Data":"5a71cd12e1e75e01025e2d73495a7bff00a4dc5cbd1e49f981c7ce41e2e078d9"} Sep 11 14:17:18.805493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.804316 2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32273522-7dd5-42c8-b2ce-fb3c46906000-ca-trust-extracted\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:17:18.805493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.804331 2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32273522-7dd5-42c8-b2ce-fb3c46906000-bound-sa-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:17:18.805493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.804337 2573 scope.go:117] "RemoveContainer" containerID="a4b0665fd4e2df8a70b62645b517c603927b68e23fc03ffcd539a76aeb2470e1" Sep 11 14:17:18.805493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.804345 2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32273522-7dd5-42c8-b2ce-fb3c46906000-registry-certificates\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:17:18.815603 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.815334 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-6qbfq" Sep 11 14:17:18.828439 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.826669 2573 scope.go:117] "RemoveContainer" containerID="a4b0665fd4e2df8a70b62645b517c603927b68e23fc03ffcd539a76aeb2470e1" Sep 11 14:17:18.828439 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:17:18.827307 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b0665fd4e2df8a70b62645b517c603927b68e23fc03ffcd539a76aeb2470e1\": container with ID starting with a4b0665fd4e2df8a70b62645b517c603927b68e23fc03ffcd539a76aeb2470e1 not found: ID does not exist" containerID="a4b0665fd4e2df8a70b62645b517c603927b68e23fc03ffcd539a76aeb2470e1" Sep 11 14:17:18.828439 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.827347 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b0665fd4e2df8a70b62645b517c603927b68e23fc03ffcd539a76aeb2470e1"} err="failed to get container status \"a4b0665fd4e2df8a70b62645b517c603927b68e23fc03ffcd539a76aeb2470e1\": rpc error: code = NotFound desc = could not find container \"a4b0665fd4e2df8a70b62645b517c603927b68e23fc03ffcd539a76aeb2470e1\": container with ID starting with a4b0665fd4e2df8a70b62645b517c603927b68e23fc03ffcd539a76aeb2470e1 not found: ID does not exist" Sep 11 14:17:18.895562 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.895520 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-856c4b595b-4k97s"] Sep 11 14:17:18.907762 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:18.907731 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-856c4b595b-4k97s"] Sep 11 14:17:19.053396 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:19.053365 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b4d6f4ddb-mn4gk"] Sep 11 14:17:19.055551 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:17:19.055521 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd05741_42cc_45aa_a525_f0f24dce2105.slice/crio-e72894cb281d5325ceb8cde622b0b1ba36c732ed6ae3d28fe851427c48755e30 WatchSource:0}: Error finding container e72894cb281d5325ceb8cde622b0b1ba36c732ed6ae3d28fe851427c48755e30: Status 404 returned error can't find the container with id e72894cb281d5325ceb8cde622b0b1ba36c732ed6ae3d28fe851427c48755e30 Sep 11 14:17:19.815435 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:19.815390 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4d6f4ddb-mn4gk" event={"ID":"9cd05741-42cc-45aa-a525-f0f24dce2105","Type":"ContainerStarted","Data":"e72894cb281d5325ceb8cde622b0b1ba36c732ed6ae3d28fe851427c48755e30"} Sep 11 14:17:20.190525 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:20.190171 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32273522-7dd5-42c8-b2ce-fb3c46906000" path="/var/lib/kubelet/pods/32273522-7dd5-42c8-b2ce-fb3c46906000/volumes" Sep 11 14:17:22.835911 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:22.835876 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4d6f4ddb-mn4gk" event={"ID":"9cd05741-42cc-45aa-a525-f0f24dce2105","Type":"ContainerStarted","Data":"770efef858f2f68c6a0a2f8dd687f0abf3da868e4484ef2451e47a6104cd95ed"} Sep 11 14:17:22.928954 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:22.928855 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b4d6f4ddb-mn4gk" podStartSLOduration=1.33414906 podStartE2EDuration="4.928819137s" totalImagesPullingTime="3.594670077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:22.927953526 +0000 UTC m=+155.264921718" watchObservedRunningTime="2026-09-11 14:17:22.928819137 +0000 UTC m=+155.265787328" Sep 11 14:17:25.680571 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:25.680532 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:17:25.684925 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:25.684901 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-546c45558c-vlls4" Sep 11 14:17:28.718861 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:28.718813 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:28.719401 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:28.719115 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:28.723774 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:28.723749 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:17:28.866846 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:17:28.866803 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:18:01.575056 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:01.575019 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:18:01.575475 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:01.575322 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="alertmanager" containerID="cri-o://07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82" gracePeriod=120 Sep 11 14:18:01.575475 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:01.575422 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="kube-rbac-proxy-web" containerID="cri-o://786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9" gracePeriod=120 Sep 11 14:18:01.575599 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:01.575456 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="kube-rbac-proxy-metric" containerID="cri-o://59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4" gracePeriod=120 Sep 11 14:18:01.575599 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:01.575480 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="config-reloader" containerID="cri-o://f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1" gracePeriod=120 Sep 11 14:18:01.575599 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:01.575481 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="kube-rbac-proxy" containerID="cri-o://830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192" gracePeriod=120 Sep 11 14:18:01.575599 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:01.575448 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="prom-label-proxy" containerID="cri-o://5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311" gracePeriod=120 Sep 11 14:18:02.005932 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.005897 2573 generic.go:356] "Generic (PLEG): container finished" podID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerID="5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311" exitCode=0 Sep 11 14:18:02.005932 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.005926 2573 generic.go:356] "Generic (PLEG): container finished" podID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerID="830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192" exitCode=0 Sep 11 14:18:02.005932 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.005933 2573 generic.go:356] "Generic (PLEG): container finished" podID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerID="f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1" exitCode=0 Sep 11 14:18:02.005932 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.005940 2573 generic.go:356] "Generic (PLEG): container finished" podID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerID="07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82" exitCode=0 Sep 11 14:18:02.006181 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.005979 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0237a6f-fc53-4839-8cb1-8803c6ee41a8","Type":"ContainerDied","Data":"5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311"} Sep 11 14:18:02.006181 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.006069 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0237a6f-fc53-4839-8cb1-8803c6ee41a8","Type":"ContainerDied","Data":"830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192"} Sep 11 14:18:02.006181 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.006097 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0237a6f-fc53-4839-8cb1-8803c6ee41a8","Type":"ContainerDied","Data":"f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1"} Sep 11 14:18:02.006181 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.006121 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0237a6f-fc53-4839-8cb1-8803c6ee41a8","Type":"ContainerDied","Data":"07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82"} Sep 11 14:18:02.862711 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.862687 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:02.979403 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.979321 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cp7j\" (UniqueName: \"kubernetes.io/projected/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-kube-api-access-2cp7j\") pod \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " Sep 11 14:18:02.979403 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.979371 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-config-volume\") pod \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " Sep 11 14:18:02.979610 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.979405 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-config-out\") pod \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " Sep 11 14:18:02.979610 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.979446 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-alertmanager-trusted-ca-bundle\") pod \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " Sep 11 14:18:02.979610 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.979475 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-alertmanager-main-db\") pod \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " Sep 11 14:18:02.979610 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.979501 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-tls-assets\") pod \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " Sep 11 14:18:02.979610 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.979539 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-metrics-client-ca\") pod \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " Sep 11 14:18:02.979610 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.979572 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " Sep 11 14:18:02.979932 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.979883 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d0237a6f-fc53-4839-8cb1-8803c6ee41a8" (UID: "d0237a6f-fc53-4839-8cb1-8803c6ee41a8"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:18:02.979932 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.979911 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d0237a6f-fc53-4839-8cb1-8803c6ee41a8" (UID: "d0237a6f-fc53-4839-8cb1-8803c6ee41a8"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:02.980046 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.979980 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d0237a6f-fc53-4839-8cb1-8803c6ee41a8" (UID: "d0237a6f-fc53-4839-8cb1-8803c6ee41a8"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:02.980100 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.980048 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy-web\") pod \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " Sep 11 14:18:02.980680 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.980082 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-main-tls\") pod \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " Sep 11 14:18:02.980680 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.980211 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-web-config\") pod \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " Sep 11 14:18:02.980680 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.980255 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-cluster-tls-config\") pod \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " Sep 11 14:18:02.980680 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.980285 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy\") pod \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\" (UID: \"d0237a6f-fc53-4839-8cb1-8803c6ee41a8\") " Sep 11 14:18:02.981167 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.981101 2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:02.981167 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.981129 2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-alertmanager-main-db\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:02.981167 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.981147 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-metrics-client-ca\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:02.982437 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.982396 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d0237a6f-fc53-4839-8cb1-8803c6ee41a8" (UID: "d0237a6f-fc53-4839-8cb1-8803c6ee41a8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:02.982537 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.982441 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-config-volume" (OuterVolumeSpecName: "config-volume") pod "d0237a6f-fc53-4839-8cb1-8803c6ee41a8" (UID: "d0237a6f-fc53-4839-8cb1-8803c6ee41a8"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:02.982537 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.982471 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-kube-api-access-2cp7j" (OuterVolumeSpecName: "kube-api-access-2cp7j") pod "d0237a6f-fc53-4839-8cb1-8803c6ee41a8" (UID: "d0237a6f-fc53-4839-8cb1-8803c6ee41a8"). InnerVolumeSpecName "kube-api-access-2cp7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:18:02.982638 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.982572 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-config-out" (OuterVolumeSpecName: "config-out") pod "d0237a6f-fc53-4839-8cb1-8803c6ee41a8" (UID: "d0237a6f-fc53-4839-8cb1-8803c6ee41a8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:18:02.982709 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.982687 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d0237a6f-fc53-4839-8cb1-8803c6ee41a8" (UID: "d0237a6f-fc53-4839-8cb1-8803c6ee41a8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:18:02.983492 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.983462 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d0237a6f-fc53-4839-8cb1-8803c6ee41a8" (UID: "d0237a6f-fc53-4839-8cb1-8803c6ee41a8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:02.983717 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.983692 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d0237a6f-fc53-4839-8cb1-8803c6ee41a8" (UID: "d0237a6f-fc53-4839-8cb1-8803c6ee41a8"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:02.983995 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.983975 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d0237a6f-fc53-4839-8cb1-8803c6ee41a8" (UID: "d0237a6f-fc53-4839-8cb1-8803c6ee41a8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:02.988421 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.988335 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d0237a6f-fc53-4839-8cb1-8803c6ee41a8" (UID: "d0237a6f-fc53-4839-8cb1-8803c6ee41a8"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:02.993691 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:02.993670 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-web-config" (OuterVolumeSpecName: "web-config") pod "d0237a6f-fc53-4839-8cb1-8803c6ee41a8" (UID: "d0237a6f-fc53-4839-8cb1-8803c6ee41a8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:03.013180 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.013152 2573 generic.go:356] "Generic (PLEG): container finished" podID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerID="59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4" exitCode=0 Sep 11 14:18:03.013180 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.013176 2573 generic.go:356] "Generic (PLEG): container finished" podID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerID="786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9" exitCode=0 Sep 11 14:18:03.013285 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.013223 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0237a6f-fc53-4839-8cb1-8803c6ee41a8","Type":"ContainerDied","Data":"59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4"} Sep 11 14:18:03.013285 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.013246 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.013357 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.013298 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0237a6f-fc53-4839-8cb1-8803c6ee41a8","Type":"ContainerDied","Data":"786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9"} Sep 11 14:18:03.013357 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.013339 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0237a6f-fc53-4839-8cb1-8803c6ee41a8","Type":"ContainerDied","Data":"87a373456b79be8d8e3aee34677bea90ac7f061825962c2da8b43e7c8fbc763d"} Sep 11 14:18:03.013413 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.013366 2573 scope.go:117] "RemoveContainer" containerID="5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311" Sep 11 14:18:03.021738 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.021709 2573 scope.go:117] "RemoveContainer" containerID="59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4" Sep 11 14:18:03.028302 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.028286 2573 scope.go:117] "RemoveContainer" containerID="830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192" Sep 11 14:18:03.034622 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.034606 2573 scope.go:117] "RemoveContainer" containerID="786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9" Sep 11 14:18:03.041656 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.041632 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:18:03.042043 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.042026 2573 scope.go:117] "RemoveContainer" containerID="f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1" Sep 11 14:18:03.046479 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.046459 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:18:03.049011 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.048993 2573 scope.go:117] "RemoveContainer" containerID="07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82" Sep 11 14:18:03.056866 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.056770 2573 scope.go:117] "RemoveContainer" containerID="8e7f3eadca4abc87861388070926bad750e5144007c1d236ed9c75fe8a9fd351" Sep 11 14:18:03.063987 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.063970 2573 scope.go:117] "RemoveContainer" containerID="5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311" Sep 11 14:18:03.064233 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:18:03.064213 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311\": container with ID starting with 5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311 not found: ID does not exist" containerID="5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311" Sep 11 14:18:03.064281 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.064241 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311"} err="failed to get container status \"5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311\": rpc error: code = NotFound desc = could not find container \"5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311\": container with ID starting with 5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311 not found: ID does not exist" Sep 11 14:18:03.064281 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.064261 2573 scope.go:117] "RemoveContainer" containerID="59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4" Sep 11 14:18:03.064490 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:18:03.064474 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4\": container with ID starting with 59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4 not found: ID does not exist" containerID="59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4" Sep 11 14:18:03.064529 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.064493 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4"} err="failed to get container status \"59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4\": rpc error: code = NotFound desc = could not find container \"59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4\": container with ID starting with 59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4 not found: ID does not exist" Sep 11 14:18:03.064529 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.064506 2573 scope.go:117] "RemoveContainer" containerID="830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192" Sep 11 14:18:03.064703 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:18:03.064688 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192\": container with ID starting with 830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192 not found: ID does not exist" containerID="830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192" Sep 11 14:18:03.064741 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.064706 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192"} err="failed to get container status \"830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192\": rpc error: code = NotFound desc = could not find container \"830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192\": container with ID starting with 830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192 not found: ID does not exist" Sep 11 14:18:03.064741 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.064719 2573 scope.go:117] "RemoveContainer" containerID="786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9" Sep 11 14:18:03.064931 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:18:03.064916 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9\": container with ID starting with 786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9 not found: ID does not exist" containerID="786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9" Sep 11 14:18:03.064975 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.064934 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9"} err="failed to get container status \"786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9\": rpc error: code = NotFound desc = could not find container \"786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9\": container with ID starting with 786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9 not found: ID does not exist" Sep 11 14:18:03.064975 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.064949 2573 scope.go:117] "RemoveContainer" containerID="f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1" Sep 11 14:18:03.065145 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:18:03.065130 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1\": container with ID starting with f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1 not found: ID does not exist" containerID="f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1" Sep 11 14:18:03.065186 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.065148 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1"} err="failed to get container status \"f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1\": rpc error: code = NotFound desc = could not find container \"f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1\": container with ID starting with f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1 not found: ID does not exist" Sep 11 14:18:03.065186 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.065160 2573 scope.go:117] "RemoveContainer" containerID="07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82" Sep 11 14:18:03.065340 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:18:03.065326 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82\": container with ID starting with 07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82 not found: ID does not exist" containerID="07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82" Sep 11 14:18:03.065382 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.065343 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82"} err="failed to get container status \"07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82\": rpc error: code = NotFound desc = could not find container \"07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82\": container with ID starting with 07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82 not found: ID does not exist" Sep 11 14:18:03.065382 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.065353 2573 scope.go:117] "RemoveContainer" containerID="8e7f3eadca4abc87861388070926bad750e5144007c1d236ed9c75fe8a9fd351" Sep 11 14:18:03.065539 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:18:03.065524 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7f3eadca4abc87861388070926bad750e5144007c1d236ed9c75fe8a9fd351\": container with ID starting with 8e7f3eadca4abc87861388070926bad750e5144007c1d236ed9c75fe8a9fd351 not found: ID does not exist" containerID="8e7f3eadca4abc87861388070926bad750e5144007c1d236ed9c75fe8a9fd351" Sep 11 14:18:03.065573 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.065541 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7f3eadca4abc87861388070926bad750e5144007c1d236ed9c75fe8a9fd351"} err="failed to get container status \"8e7f3eadca4abc87861388070926bad750e5144007c1d236ed9c75fe8a9fd351\": rpc error: code = NotFound desc = could not find container \"8e7f3eadca4abc87861388070926bad750e5144007c1d236ed9c75fe8a9fd351\": container with ID starting with 8e7f3eadca4abc87861388070926bad750e5144007c1d236ed9c75fe8a9fd351 not found: ID does not exist" Sep 11 14:18:03.065573 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.065553 2573 scope.go:117] "RemoveContainer" containerID="5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311" Sep 11 14:18:03.065730 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.065715 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311"} err="failed to get container status \"5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311\": rpc error: code = NotFound desc = could not find container \"5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311\": container with ID starting with 5fffc5cd6a1a7c198a165992ae984a62730d162ed9763ce2b0b4c600f0154311 not found: ID does not exist" Sep 11 14:18:03.065770 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.065730 2573 scope.go:117] "RemoveContainer" containerID="59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4" Sep 11 14:18:03.065948 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.065929 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4"} err="failed to get container status \"59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4\": rpc error: code = NotFound desc = could not find container \"59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4\": container with ID starting with 59b66207584b31d531e1cc75e412500b4bc10ea027fe0b0d90bc93d0682816d4 not found: ID does not exist" Sep 11 14:18:03.065994 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.065949 2573 scope.go:117] "RemoveContainer" containerID="830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192" Sep 11 14:18:03.066173 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.066156 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192"} err="failed to get container status \"830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192\": rpc error: code = NotFound desc = could not find container \"830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192\": container with ID starting with 830b047ed47a56bc0812da3eac87328c7cb9be058e2b0b5550e2877a10269192 not found: ID does not exist" Sep 11 14:18:03.066173 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.066172 2573 scope.go:117] "RemoveContainer" containerID="786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9" Sep 11 14:18:03.066346 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.066330 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9"} err="failed to get container status \"786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9\": rpc error: code = NotFound desc = could not find container \"786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9\": container with ID starting with 786c2fd5dcd13fb454d2bf30cc441c4cdc693c98e0666837bca82dd11d9455a9 not found: ID does not exist" Sep 11 14:18:03.066346 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.066346 2573 scope.go:117] "RemoveContainer" containerID="f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1" Sep 11 14:18:03.066507 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.066493 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1"} err="failed to get container status \"f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1\": rpc error: code = NotFound desc = could not find container \"f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1\": container with ID starting with f326ac8e975c16c4c4c0d843413479b552bbeb410b1c249a35f1d356ff42e8a1 not found: ID does not exist" Sep 11 14:18:03.066562 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.066506 2573 scope.go:117] "RemoveContainer" containerID="07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82" Sep 11 14:18:03.066724 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.066708 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82"} err="failed to get container status \"07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82\": rpc error: code = NotFound desc = could not find container \"07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82\": container with ID starting with 07ddec0ee0755d1f6667ef9fa44185f40d640e7cf8a932d457d563fab8c76b82 not found: ID does not exist" Sep 11 14:18:03.066724 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.066723 2573 scope.go:117] "RemoveContainer" containerID="8e7f3eadca4abc87861388070926bad750e5144007c1d236ed9c75fe8a9fd351" Sep 11 14:18:03.066930 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.066907 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7f3eadca4abc87861388070926bad750e5144007c1d236ed9c75fe8a9fd351"} err="failed to get container status \"8e7f3eadca4abc87861388070926bad750e5144007c1d236ed9c75fe8a9fd351\": rpc error: code = NotFound desc = could not find container \"8e7f3eadca4abc87861388070926bad750e5144007c1d236ed9c75fe8a9fd351\": container with ID starting with 8e7f3eadca4abc87861388070926bad750e5144007c1d236ed9c75fe8a9fd351 not found: ID does not exist" Sep 11 14:18:03.082316 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.082293 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:03.082316 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.082312 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:03.082423 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.082322 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-main-tls\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:03.082423 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.082352 2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-web-config\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:03.082423 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.082361 2573 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-cluster-tls-config\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:03.082423 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.082370 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:03.082423 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.082378 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cp7j\" (UniqueName: \"kubernetes.io/projected/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-kube-api-access-2cp7j\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:03.082423 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.082386 2573 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-config-volume\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:03.082423 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.082394 2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-config-out\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:03.082423 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.082401 2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0237a6f-fc53-4839-8cb1-8803c6ee41a8-tls-assets\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:03.087049 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087028 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:18:03.087336 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087324 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="prom-label-proxy" Sep 11 14:18:03.087377 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087338 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="prom-label-proxy" Sep 11 14:18:03.087377 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087349 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="alertmanager" Sep 11 14:18:03.087377 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087354 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="alertmanager" Sep 11 14:18:03.087377 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087363 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32273522-7dd5-42c8-b2ce-fb3c46906000" containerName="registry" Sep 11 14:18:03.087377 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087369 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="32273522-7dd5-42c8-b2ce-fb3c46906000" containerName="registry" Sep 11 14:18:03.087377 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087375 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="config-reloader" Sep 11 14:18:03.087576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087380 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="config-reloader" Sep 11 14:18:03.087576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087386 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="kube-rbac-proxy-metric" Sep 11 14:18:03.087576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087391 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="kube-rbac-proxy-metric" Sep 11 14:18:03.087576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087398 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="init-config-reloader" Sep 11 14:18:03.087576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087405 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="init-config-reloader" Sep 11 14:18:03.087576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087410 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="kube-rbac-proxy-web" Sep 11 14:18:03.087576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087416 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="kube-rbac-proxy-web" Sep 11 14:18:03.087576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087425 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="kube-rbac-proxy" Sep 11 14:18:03.087576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087430 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="kube-rbac-proxy" Sep 11 14:18:03.087576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087509 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="kube-rbac-proxy" Sep 11 14:18:03.087576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087518 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="prom-label-proxy" Sep 11 14:18:03.087576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087525 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="kube-rbac-proxy-metric" Sep 11 14:18:03.087576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087532 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="32273522-7dd5-42c8-b2ce-fb3c46906000" containerName="registry" Sep 11 14:18:03.087576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087539 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="kube-rbac-proxy-web" Sep 11 14:18:03.087576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087544 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="config-reloader" Sep 11 14:18:03.087576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.087550 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" containerName="alertmanager" Sep 11 14:18:03.124160 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.124133 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:18:03.124272 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.124254 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.138679 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.138657 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 14:18:03.138774 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.138692 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 14:18:03.138774 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.138694 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 14:18:03.138960 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.138944 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 14:18:03.139011 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.138958 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 14:18:03.139065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.139018 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 14:18:03.139065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.139044 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8l5pn\"" Sep 11 14:18:03.139065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.139047 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 14:18:03.139192 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.139018 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 14:18:03.145863 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.145825 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 14:18:03.283617 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.283535 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.283617 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.283572 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.283617 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.283597 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecc99b9f-1247-41cf-8312-556ccdb5af4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.283913 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.283641 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ecc99b9f-1247-41cf-8312-556ccdb5af4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.283913 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.283680 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.283913 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.283708 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfhq7\" (UniqueName: \"kubernetes.io/projected/ecc99b9f-1247-41cf-8312-556ccdb5af4a-kube-api-access-pfhq7\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.283913 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.283777 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.283913 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.283815 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.283913 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.283889 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-web-config\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.284172 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.283926 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ecc99b9f-1247-41cf-8312-556ccdb5af4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.284172 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.283971 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.284172 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.284012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecc99b9f-1247-41cf-8312-556ccdb5af4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.284172 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.284038 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ecc99b9f-1247-41cf-8312-556ccdb5af4a-config-out\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.384775 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.384747 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.385004 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.384780 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.385004 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.384799 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-web-config\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.385004 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.384852 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ecc99b9f-1247-41cf-8312-556ccdb5af4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.385004 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.384877 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.385004 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.384913 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecc99b9f-1247-41cf-8312-556ccdb5af4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.385004 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.384938 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ecc99b9f-1247-41cf-8312-556ccdb5af4a-config-out\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.385004 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.384964 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.385004 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.384999 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.385402 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.385031 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecc99b9f-1247-41cf-8312-556ccdb5af4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.385402 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.385054 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ecc99b9f-1247-41cf-8312-556ccdb5af4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.385402 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.385082 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.385402 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.385105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfhq7\" (UniqueName: \"kubernetes.io/projected/ecc99b9f-1247-41cf-8312-556ccdb5af4a-kube-api-access-pfhq7\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.387078 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.385704 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecc99b9f-1247-41cf-8312-556ccdb5af4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.387078 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.385997 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ecc99b9f-1247-41cf-8312-556ccdb5af4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.387078 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.386753 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecc99b9f-1247-41cf-8312-556ccdb5af4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.388211 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.387880 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ecc99b9f-1247-41cf-8312-556ccdb5af4a-config-out\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.388211 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.387989 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-web-config\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.388211 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.388044 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.388211 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.388157 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.388211 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.388163 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.388211 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.388186 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.388211 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.388208 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.389127 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.389111 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ecc99b9f-1247-41cf-8312-556ccdb5af4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.389721 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.389702 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ecc99b9f-1247-41cf-8312-556ccdb5af4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.401809 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.401787 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfhq7\" (UniqueName: \"kubernetes.io/projected/ecc99b9f-1247-41cf-8312-556ccdb5af4a-kube-api-access-pfhq7\") pod \"alertmanager-main-0\" (UID: \"ecc99b9f-1247-41cf-8312-556ccdb5af4a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.434844 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.434806 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:03.724914 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:03.724884 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:18:03.728710 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:18:03.728686 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc99b9f_1247_41cf_8312_556ccdb5af4a.slice/crio-10bf47391b06cbd67a3bc0bbadc287720074950697d9b383ceef992063f4247b WatchSource:0}: Error finding container 10bf47391b06cbd67a3bc0bbadc287720074950697d9b383ceef992063f4247b: Status 404 returned error can't find the container with id 10bf47391b06cbd67a3bc0bbadc287720074950697d9b383ceef992063f4247b Sep 11 14:18:04.019548 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:04.019458 2573 generic.go:356] "Generic (PLEG): container finished" podID="ecc99b9f-1247-41cf-8312-556ccdb5af4a" containerID="6f6e9dc3824c7b06ceaac0aa46defa1bda074947dd1b94571310d40b60fc6429" exitCode=0 Sep 11 14:18:04.019548 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:04.019531 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecc99b9f-1247-41cf-8312-556ccdb5af4a","Type":"ContainerDied","Data":"6f6e9dc3824c7b06ceaac0aa46defa1bda074947dd1b94571310d40b60fc6429"} Sep 11 14:18:04.020024 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:04.019599 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecc99b9f-1247-41cf-8312-556ccdb5af4a","Type":"ContainerStarted","Data":"10bf47391b06cbd67a3bc0bbadc287720074950697d9b383ceef992063f4247b"} Sep 11 14:18:04.187251 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:04.187220 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0237a6f-fc53-4839-8cb1-8803c6ee41a8" path="/var/lib/kubelet/pods/d0237a6f-fc53-4839-8cb1-8803c6ee41a8/volumes" Sep 11 14:18:05.028121 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:05.028029 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecc99b9f-1247-41cf-8312-556ccdb5af4a","Type":"ContainerStarted","Data":"bbefdd8f418218fd36683e36cf475b4ec769ee2158a8482d6143016c5ab795d1"} Sep 11 14:18:05.028121 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:05.028094 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecc99b9f-1247-41cf-8312-556ccdb5af4a","Type":"ContainerStarted","Data":"98b8d9105195e8d58187b61f5cac1cb15a10d690a065241e2642a219a5bb31da"} Sep 11 14:18:05.028121 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:05.028121 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecc99b9f-1247-41cf-8312-556ccdb5af4a","Type":"ContainerStarted","Data":"855097e5a2a9c2e8acb47e6b1500c6e39da35d5e668bc81a88ec8551ef162b25"} Sep 11 14:18:05.028575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:05.028160 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecc99b9f-1247-41cf-8312-556ccdb5af4a","Type":"ContainerStarted","Data":"9c0fb14486c9ba50e65bbebf8c253178378b168a99c06f4c77775b7c52ce99d0"} Sep 11 14:18:05.028575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:05.028189 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecc99b9f-1247-41cf-8312-556ccdb5af4a","Type":"ContainerStarted","Data":"1b18bd83ca6de28a0c5c0c21a6cb92486d2d24bc4e9cf938dd4a7b879418540d"} Sep 11 14:18:05.028575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:05.028214 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecc99b9f-1247-41cf-8312-556ccdb5af4a","Type":"ContainerStarted","Data":"61e2a4d7dabde9675053a9454bafead813fad1190b56aa742d2897db6e43ef72"} Sep 11 14:18:05.080260 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:05.080210 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.06520996 podStartE2EDuration="2.080193801s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.983841ms" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:05.078596865 +0000 UTC m=+197.415565056" watchObservedRunningTime="2026-09-11 14:18:05.080193801 +0000 UTC m=+197.417161991" Sep 11 14:18:11.822407 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:11.822375 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b4d6f4ddb-mn4gk"] Sep 11 14:18:36.840897 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:36.840853 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b4d6f4ddb-mn4gk" podUID="9cd05741-42cc-45aa-a525-f0f24dce2105" containerName="console" containerID="cri-o://770efef858f2f68c6a0a2f8dd687f0abf3da868e4484ef2451e47a6104cd95ed" gracePeriod=15 Sep 11 14:18:37.094444 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.094394 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:18:37.159377 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.159347 2573 generic.go:356] "Generic (PLEG): container finished" podID="9cd05741-42cc-45aa-a525-f0f24dce2105" containerID="770efef858f2f68c6a0a2f8dd687f0abf3da868e4484ef2451e47a6104cd95ed" exitCode=0 Sep 11 14:18:37.159517 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.159389 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4d6f4ddb-mn4gk" Sep 11 14:18:37.159517 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.159389 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4d6f4ddb-mn4gk" event={"ID":"9cd05741-42cc-45aa-a525-f0f24dce2105","Type":"ContainerDied","Data":"770efef858f2f68c6a0a2f8dd687f0abf3da868e4484ef2451e47a6104cd95ed"} Sep 11 14:18:37.159517 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.159498 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4d6f4ddb-mn4gk" event={"ID":"9cd05741-42cc-45aa-a525-f0f24dce2105","Type":"ContainerDied","Data":"e72894cb281d5325ceb8cde622b0b1ba36c732ed6ae3d28fe851427c48755e30"} Sep 11 14:18:37.159621 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.159522 2573 scope.go:117] "RemoveContainer" containerID="770efef858f2f68c6a0a2f8dd687f0abf3da868e4484ef2451e47a6104cd95ed" Sep 11 14:18:37.167299 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.167284 2573 scope.go:117] "RemoveContainer" containerID="770efef858f2f68c6a0a2f8dd687f0abf3da868e4484ef2451e47a6104cd95ed" Sep 11 14:18:37.167579 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:18:37.167558 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"770efef858f2f68c6a0a2f8dd687f0abf3da868e4484ef2451e47a6104cd95ed\": container with ID starting with 770efef858f2f68c6a0a2f8dd687f0abf3da868e4484ef2451e47a6104cd95ed not found: ID does not exist" containerID="770efef858f2f68c6a0a2f8dd687f0abf3da868e4484ef2451e47a6104cd95ed" Sep 11 14:18:37.167630 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.167587 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770efef858f2f68c6a0a2f8dd687f0abf3da868e4484ef2451e47a6104cd95ed"} err="failed to get container status \"770efef858f2f68c6a0a2f8dd687f0abf3da868e4484ef2451e47a6104cd95ed\": rpc error: code = NotFound desc = could not find container \"770efef858f2f68c6a0a2f8dd687f0abf3da868e4484ef2451e47a6104cd95ed\": container with ID starting with 770efef858f2f68c6a0a2f8dd687f0abf3da868e4484ef2451e47a6104cd95ed not found: ID does not exist" Sep 11 14:18:37.170864 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.170847 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-service-ca\") pod \"9cd05741-42cc-45aa-a525-f0f24dce2105\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " Sep 11 14:18:37.170920 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.170878 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-trusted-ca-bundle\") pod \"9cd05741-42cc-45aa-a525-f0f24dce2105\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " Sep 11 14:18:37.170976 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.170920 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-console-config\") pod \"9cd05741-42cc-45aa-a525-f0f24dce2105\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " Sep 11 14:18:37.170976 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.170938 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzf7b\" (UniqueName: \"kubernetes.io/projected/9cd05741-42cc-45aa-a525-f0f24dce2105-kube-api-access-rzf7b\") pod \"9cd05741-42cc-45aa-a525-f0f24dce2105\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " Sep 11 14:18:37.170976 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.170973 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cd05741-42cc-45aa-a525-f0f24dce2105-console-oauth-config\") pod \"9cd05741-42cc-45aa-a525-f0f24dce2105\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " Sep 11 14:18:37.171120 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.170997 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-oauth-serving-cert\") pod \"9cd05741-42cc-45aa-a525-f0f24dce2105\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " Sep 11 14:18:37.171120 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.171013 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cd05741-42cc-45aa-a525-f0f24dce2105-console-serving-cert\") pod \"9cd05741-42cc-45aa-a525-f0f24dce2105\" (UID: \"9cd05741-42cc-45aa-a525-f0f24dce2105\") " Sep 11 14:18:37.171320 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.171290 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-service-ca" (OuterVolumeSpecName: "service-ca") pod "9cd05741-42cc-45aa-a525-f0f24dce2105" (UID: "9cd05741-42cc-45aa-a525-f0f24dce2105"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:37.171438 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.171334 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9cd05741-42cc-45aa-a525-f0f24dce2105" (UID: "9cd05741-42cc-45aa-a525-f0f24dce2105"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:37.171438 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.171349 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-console-config" (OuterVolumeSpecName: "console-config") pod "9cd05741-42cc-45aa-a525-f0f24dce2105" (UID: "9cd05741-42cc-45aa-a525-f0f24dce2105"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:37.171438 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.171393 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9cd05741-42cc-45aa-a525-f0f24dce2105" (UID: "9cd05741-42cc-45aa-a525-f0f24dce2105"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:37.173209 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.173180 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd05741-42cc-45aa-a525-f0f24dce2105-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9cd05741-42cc-45aa-a525-f0f24dce2105" (UID: "9cd05741-42cc-45aa-a525-f0f24dce2105"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:37.173283 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.173220 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd05741-42cc-45aa-a525-f0f24dce2105-kube-api-access-rzf7b" (OuterVolumeSpecName: "kube-api-access-rzf7b") pod "9cd05741-42cc-45aa-a525-f0f24dce2105" (UID: "9cd05741-42cc-45aa-a525-f0f24dce2105"). InnerVolumeSpecName "kube-api-access-rzf7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:18:37.173283 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.173225 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd05741-42cc-45aa-a525-f0f24dce2105-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9cd05741-42cc-45aa-a525-f0f24dce2105" (UID: "9cd05741-42cc-45aa-a525-f0f24dce2105"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:37.272323 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.272281 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-service-ca\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:37.272323 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.272316 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-trusted-ca-bundle\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:37.272502 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.272331 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-console-config\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:37.272502 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.272347 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzf7b\" (UniqueName: \"kubernetes.io/projected/9cd05741-42cc-45aa-a525-f0f24dce2105-kube-api-access-rzf7b\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:37.272502 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.272362 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cd05741-42cc-45aa-a525-f0f24dce2105-console-oauth-config\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:37.272502 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.272375 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cd05741-42cc-45aa-a525-f0f24dce2105-oauth-serving-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:37.272502 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.272389 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cd05741-42cc-45aa-a525-f0f24dce2105-console-serving-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:18:37.487007 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.486979 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b4d6f4ddb-mn4gk"] Sep 11 14:18:37.491856 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:37.491817 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b4d6f4ddb-mn4gk"] Sep 11 14:18:38.191699 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:38.191640 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cd05741-42cc-45aa-a525-f0f24dce2105" path="/var/lib/kubelet/pods/9cd05741-42cc-45aa-a525-f0f24dce2105/volumes" Sep 11 14:18:51.257929 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.257898 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2"] Sep 11 14:18:51.258320 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.258196 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd05741-42cc-45aa-a525-f0f24dce2105" containerName="console" Sep 11 14:18:51.258320 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.258208 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd05741-42cc-45aa-a525-f0f24dce2105" containerName="console" Sep 11 14:18:51.258320 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.258272 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd05741-42cc-45aa-a525-f0f24dce2105" containerName="console" Sep 11 14:18:51.261126 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.261108 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" Sep 11 14:18:51.265630 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.265605 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4brs6\"" Sep 11 14:18:51.265726 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.265609 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:18:51.266642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.266624 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:18:51.391342 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.391308 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6z7c\" (UniqueName: \"kubernetes.io/projected/a40685a2-52cb-4256-85b9-ca92539ea4a7-kube-api-access-m6z7c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2\" (UID: \"a40685a2-52cb-4256-85b9-ca92539ea4a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" Sep 11 14:18:51.391342 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.391343 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a40685a2-52cb-4256-85b9-ca92539ea4a7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2\" (UID: \"a40685a2-52cb-4256-85b9-ca92539ea4a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" Sep 11 14:18:51.391513 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.391364 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a40685a2-52cb-4256-85b9-ca92539ea4a7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2\" (UID: \"a40685a2-52cb-4256-85b9-ca92539ea4a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" Sep 11 14:18:51.400465 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.400435 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2"] Sep 11 14:18:51.492042 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.492010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a40685a2-52cb-4256-85b9-ca92539ea4a7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2\" (UID: \"a40685a2-52cb-4256-85b9-ca92539ea4a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" Sep 11 14:18:51.492183 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.492091 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6z7c\" (UniqueName: \"kubernetes.io/projected/a40685a2-52cb-4256-85b9-ca92539ea4a7-kube-api-access-m6z7c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2\" (UID: \"a40685a2-52cb-4256-85b9-ca92539ea4a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" Sep 11 14:18:51.492183 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.492110 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a40685a2-52cb-4256-85b9-ca92539ea4a7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2\" (UID: \"a40685a2-52cb-4256-85b9-ca92539ea4a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" Sep 11 14:18:51.492427 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.492405 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a40685a2-52cb-4256-85b9-ca92539ea4a7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2\" (UID: \"a40685a2-52cb-4256-85b9-ca92539ea4a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" Sep 11 14:18:51.492427 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.492421 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a40685a2-52cb-4256-85b9-ca92539ea4a7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2\" (UID: \"a40685a2-52cb-4256-85b9-ca92539ea4a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" Sep 11 14:18:51.506109 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.506082 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6z7c\" (UniqueName: \"kubernetes.io/projected/a40685a2-52cb-4256-85b9-ca92539ea4a7-kube-api-access-m6z7c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2\" (UID: \"a40685a2-52cb-4256-85b9-ca92539ea4a7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" Sep 11 14:18:51.569493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.569441 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" Sep 11 14:18:51.863619 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:51.863569 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2"] Sep 11 14:18:51.866170 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:18:51.866145 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40685a2_52cb_4256_85b9_ca92539ea4a7.slice/crio-eaf44801f00d2febb2772bdea2c1e3a2096ad133eb324d3bac6b79f4c6228b84 WatchSource:0}: Error finding container eaf44801f00d2febb2772bdea2c1e3a2096ad133eb324d3bac6b79f4c6228b84: Status 404 returned error can't find the container with id eaf44801f00d2febb2772bdea2c1e3a2096ad133eb324d3bac6b79f4c6228b84 Sep 11 14:18:52.214716 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:52.214636 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" event={"ID":"a40685a2-52cb-4256-85b9-ca92539ea4a7","Type":"ContainerStarted","Data":"eaf44801f00d2febb2772bdea2c1e3a2096ad133eb324d3bac6b79f4c6228b84"} Sep 11 14:18:58.242190 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:58.242158 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" event={"ID":"a40685a2-52cb-4256-85b9-ca92539ea4a7","Type":"ContainerStarted","Data":"4cde5ddaa821d295d7f0482390e59c111910e0a53c6a7437300290b7bb9d0303"} Sep 11 14:18:59.247289 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:59.247256 2573 generic.go:356] "Generic (PLEG): container finished" podID="a40685a2-52cb-4256-85b9-ca92539ea4a7" containerID="4cde5ddaa821d295d7f0482390e59c111910e0a53c6a7437300290b7bb9d0303" exitCode=0 Sep 11 14:18:59.247664 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:18:59.247327 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" event={"ID":"a40685a2-52cb-4256-85b9-ca92539ea4a7","Type":"ContainerDied","Data":"4cde5ddaa821d295d7f0482390e59c111910e0a53c6a7437300290b7bb9d0303"} Sep 11 14:19:02.265969 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:02.265921 2573 generic.go:356] "Generic (PLEG): container finished" podID="a40685a2-52cb-4256-85b9-ca92539ea4a7" containerID="72828edfa74d6c4568ff3036fe69c18778c13d4f9aa8eef6202444a6b6c61a48" exitCode=0 Sep 11 14:19:02.266331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:02.265998 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" event={"ID":"a40685a2-52cb-4256-85b9-ca92539ea4a7","Type":"ContainerDied","Data":"72828edfa74d6c4568ff3036fe69c18778c13d4f9aa8eef6202444a6b6c61a48"} Sep 11 14:19:09.298659 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:09.298626 2573 generic.go:356] "Generic (PLEG): container finished" podID="a40685a2-52cb-4256-85b9-ca92539ea4a7" containerID="4d16e87b22fc349277960e9024e0b925df1fd13e1937086d4ccc2d71b4dd042b" exitCode=0 Sep 11 14:19:09.299049 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:09.298672 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" event={"ID":"a40685a2-52cb-4256-85b9-ca92539ea4a7","Type":"ContainerDied","Data":"4d16e87b22fc349277960e9024e0b925df1fd13e1937086d4ccc2d71b4dd042b"} Sep 11 14:19:10.432765 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:10.432729 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" Sep 11 14:19:10.444512 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:10.444487 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a40685a2-52cb-4256-85b9-ca92539ea4a7-bundle\") pod \"a40685a2-52cb-4256-85b9-ca92539ea4a7\" (UID: \"a40685a2-52cb-4256-85b9-ca92539ea4a7\") " Sep 11 14:19:10.444622 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:10.444548 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6z7c\" (UniqueName: \"kubernetes.io/projected/a40685a2-52cb-4256-85b9-ca92539ea4a7-kube-api-access-m6z7c\") pod \"a40685a2-52cb-4256-85b9-ca92539ea4a7\" (UID: \"a40685a2-52cb-4256-85b9-ca92539ea4a7\") " Sep 11 14:19:10.444622 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:10.444615 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a40685a2-52cb-4256-85b9-ca92539ea4a7-util\") pod \"a40685a2-52cb-4256-85b9-ca92539ea4a7\" (UID: \"a40685a2-52cb-4256-85b9-ca92539ea4a7\") " Sep 11 14:19:10.445187 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:10.445147 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40685a2-52cb-4256-85b9-ca92539ea4a7-bundle" (OuterVolumeSpecName: "bundle") pod "a40685a2-52cb-4256-85b9-ca92539ea4a7" (UID: "a40685a2-52cb-4256-85b9-ca92539ea4a7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:10.446971 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:10.446949 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40685a2-52cb-4256-85b9-ca92539ea4a7-kube-api-access-m6z7c" (OuterVolumeSpecName: "kube-api-access-m6z7c") pod "a40685a2-52cb-4256-85b9-ca92539ea4a7" (UID: "a40685a2-52cb-4256-85b9-ca92539ea4a7"). InnerVolumeSpecName "kube-api-access-m6z7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:10.450161 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:10.450128 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40685a2-52cb-4256-85b9-ca92539ea4a7-util" (OuterVolumeSpecName: "util") pod "a40685a2-52cb-4256-85b9-ca92539ea4a7" (UID: "a40685a2-52cb-4256-85b9-ca92539ea4a7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:10.545874 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:10.545815 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a40685a2-52cb-4256-85b9-ca92539ea4a7-bundle\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:19:10.545874 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:10.545860 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6z7c\" (UniqueName: \"kubernetes.io/projected/a40685a2-52cb-4256-85b9-ca92539ea4a7-kube-api-access-m6z7c\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:19:10.545874 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:10.545871 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a40685a2-52cb-4256-85b9-ca92539ea4a7-util\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:19:11.308175 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:11.308147 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" Sep 11 14:19:11.308175 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:11.308169 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rjvs2" event={"ID":"a40685a2-52cb-4256-85b9-ca92539ea4a7","Type":"ContainerDied","Data":"eaf44801f00d2febb2772bdea2c1e3a2096ad133eb324d3bac6b79f4c6228b84"} Sep 11 14:19:11.308374 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:11.308210 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaf44801f00d2febb2772bdea2c1e3a2096ad133eb324d3bac6b79f4c6228b84" Sep 11 14:19:21.646899 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:21.646866 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h6nr7"] Sep 11 14:19:21.647361 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:21.647206 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a40685a2-52cb-4256-85b9-ca92539ea4a7" containerName="pull" Sep 11 14:19:21.647361 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:21.647218 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a40685a2-52cb-4256-85b9-ca92539ea4a7" containerName="pull" Sep 11 14:19:21.647361 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:21.647227 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a40685a2-52cb-4256-85b9-ca92539ea4a7" containerName="util" Sep 11 14:19:21.647361 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:21.647232 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a40685a2-52cb-4256-85b9-ca92539ea4a7" containerName="util" Sep 11 14:19:21.647361 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:21.647251 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a40685a2-52cb-4256-85b9-ca92539ea4a7" containerName="extract" Sep 11 14:19:21.647361 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:21.647257 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a40685a2-52cb-4256-85b9-ca92539ea4a7" containerName="extract" Sep 11 14:19:21.647361 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:21.647304 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a40685a2-52cb-4256-85b9-ca92539ea4a7" containerName="extract" Sep 11 14:19:21.649259 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:21.649239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h6nr7" Sep 11 14:19:21.659379 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:19:21.659348 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"cert-manager-operator-controller-manager-55c444bf86-h6nr7\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"cert-manager-operator\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" podUID="adf44838-674f-445b-9379-ab8b015ed898" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h6nr7" Sep 11 14:19:21.659494 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:19:21.659366 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"cert-manager-operator\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 11 14:19:21.659494 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:19:21.659372 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cert-manager-operator-controller-manager-dockercfg-fzlvx\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"cert-manager-operator\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fzlvx\"" type="*v1.Secret" Sep 11 14:19:21.659847 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:19:21.659806 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"cert-manager-operator\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 11 14:19:21.724978 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:21.724949 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h6nr7"] Sep 11 14:19:21.733382 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:21.733360 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4nzm\" (UniqueName: \"kubernetes.io/projected/adf44838-674f-445b-9379-ab8b015ed898-kube-api-access-s4nzm\") pod \"cert-manager-operator-controller-manager-55c444bf86-h6nr7\" (UID: \"adf44838-674f-445b-9379-ab8b015ed898\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h6nr7" Sep 11 14:19:21.733480 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:21.733390 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adf44838-674f-445b-9379-ab8b015ed898-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-h6nr7\" (UID: \"adf44838-674f-445b-9379-ab8b015ed898\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h6nr7" Sep 11 14:19:21.834080 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:21.834051 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4nzm\" (UniqueName: \"kubernetes.io/projected/adf44838-674f-445b-9379-ab8b015ed898-kube-api-access-s4nzm\") pod \"cert-manager-operator-controller-manager-55c444bf86-h6nr7\" (UID: \"adf44838-674f-445b-9379-ab8b015ed898\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h6nr7" Sep 11 14:19:21.834205 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:21.834084 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adf44838-674f-445b-9379-ab8b015ed898-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-h6nr7\" (UID: \"adf44838-674f-445b-9379-ab8b015ed898\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h6nr7" Sep 11 14:19:21.834391 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:21.834376 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adf44838-674f-445b-9379-ab8b015ed898-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-h6nr7\" (UID: \"adf44838-674f-445b-9379-ab8b015ed898\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h6nr7" Sep 11 14:19:22.564400 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:22.564366 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:19:22.744528 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:22.744496 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 11 14:19:22.754582 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:22.754557 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4nzm\" (UniqueName: \"kubernetes.io/projected/adf44838-674f-445b-9379-ab8b015ed898-kube-api-access-s4nzm\") pod \"cert-manager-operator-controller-manager-55c444bf86-h6nr7\" (UID: \"adf44838-674f-445b-9379-ab8b015ed898\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h6nr7" Sep 11 14:19:22.840938 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:22.840913 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fzlvx\"" Sep 11 14:19:22.857524 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:22.857498 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h6nr7" Sep 11 14:19:23.152243 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:23.152213 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h6nr7"] Sep 11 14:19:23.152595 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:19:23.152570 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf44838_674f_445b_9379_ab8b015ed898.slice/crio-6fed329409d2746b784adc6496ff551a26886b5c7bb11158c41017278cdddaa6 WatchSource:0}: Error finding container 6fed329409d2746b784adc6496ff551a26886b5c7bb11158c41017278cdddaa6: Status 404 returned error can't find the container with id 6fed329409d2746b784adc6496ff551a26886b5c7bb11158c41017278cdddaa6 Sep 11 14:19:23.355227 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:23.355196 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h6nr7" event={"ID":"adf44838-674f-445b-9379-ab8b015ed898","Type":"ContainerStarted","Data":"6fed329409d2746b784adc6496ff551a26886b5c7bb11158c41017278cdddaa6"} Sep 11 14:19:25.365913 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:25.365875 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h6nr7" event={"ID":"adf44838-674f-445b-9379-ab8b015ed898","Type":"ContainerStarted","Data":"064fc7b1fbbd8a525cc2764237fb9441dd7e0a751124288a1e3e06ffb319642e"} Sep 11 14:19:25.402957 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:25.402900 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h6nr7" podStartSLOduration=2.8268648880000002 podStartE2EDuration="4.402880475s" totalImagesPullingTime="1.576015587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:25.401045508 +0000 UTC m=+277.738013712" watchObservedRunningTime="2026-09-11 14:19:25.402880475 +0000 UTC m=+277.739848667" Sep 11 14:19:26.583400 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:26.583370 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw"] Sep 11 14:19:26.586693 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:26.586678 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" Sep 11 14:19:26.595717 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:19:26.595688 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" podUID="7dc26222-e5c6-4fac-8397-135f60eb8138" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" Sep 11 14:19:26.595868 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:26.595849 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4brs6\"" Sep 11 14:19:26.595950 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:26.595891 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:19:26.599207 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:26.599189 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:19:26.672357 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:26.672328 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw"] Sep 11 14:19:26.674946 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:26.674905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7dc26222-e5c6-4fac-8397-135f60eb8138-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw\" (UID: \"7dc26222-e5c6-4fac-8397-135f60eb8138\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" Sep 11 14:19:26.675046 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:26.674992 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7dc26222-e5c6-4fac-8397-135f60eb8138-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw\" (UID: \"7dc26222-e5c6-4fac-8397-135f60eb8138\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" Sep 11 14:19:26.675046 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:26.675035 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xhsx\" (UniqueName: \"kubernetes.io/projected/7dc26222-e5c6-4fac-8397-135f60eb8138-kube-api-access-8xhsx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw\" (UID: \"7dc26222-e5c6-4fac-8397-135f60eb8138\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" Sep 11 14:19:26.775745 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:26.775672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7dc26222-e5c6-4fac-8397-135f60eb8138-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw\" (UID: \"7dc26222-e5c6-4fac-8397-135f60eb8138\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" Sep 11 14:19:26.775745 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:26.775711 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xhsx\" (UniqueName: \"kubernetes.io/projected/7dc26222-e5c6-4fac-8397-135f60eb8138-kube-api-access-8xhsx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw\" (UID: \"7dc26222-e5c6-4fac-8397-135f60eb8138\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" Sep 11 14:19:26.776140 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:26.775753 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7dc26222-e5c6-4fac-8397-135f60eb8138-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw\" (UID: \"7dc26222-e5c6-4fac-8397-135f60eb8138\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" Sep 11 14:19:26.776140 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:26.776051 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7dc26222-e5c6-4fac-8397-135f60eb8138-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw\" (UID: \"7dc26222-e5c6-4fac-8397-135f60eb8138\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" Sep 11 14:19:26.776140 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:26.776082 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7dc26222-e5c6-4fac-8397-135f60eb8138-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw\" (UID: \"7dc26222-e5c6-4fac-8397-135f60eb8138\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" Sep 11 14:19:26.811961 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:26.811933 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xhsx\" (UniqueName: \"kubernetes.io/projected/7dc26222-e5c6-4fac-8397-135f60eb8138-kube-api-access-8xhsx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw\" (UID: \"7dc26222-e5c6-4fac-8397-135f60eb8138\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" Sep 11 14:19:26.895620 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:26.895594 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" Sep 11 14:19:27.194878 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.194843 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw"] Sep 11 14:19:27.200277 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:19:27.200251 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc26222_e5c6_4fac_8397_135f60eb8138.slice/crio-3230c824b767f74a8d1d5aface0f3d2e666c83771aea76f3fcef191ade404743 WatchSource:0}: Error finding container 3230c824b767f74a8d1d5aface0f3d2e666c83771aea76f3fcef191ade404743: Status 404 returned error can't find the container with id 3230c824b767f74a8d1d5aface0f3d2e666c83771aea76f3fcef191ade404743 Sep 11 14:19:27.375382 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.375349 2573 generic.go:356] "Generic (PLEG): container finished" podID="7dc26222-e5c6-4fac-8397-135f60eb8138" containerID="b02efe97754d361b65a180fac8cb4158862fe99684e9abf8cc6c88571747453e" exitCode=0 Sep 11 14:19:27.375525 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.375390 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" event={"ID":"7dc26222-e5c6-4fac-8397-135f60eb8138","Type":"ContainerDied","Data":"b02efe97754d361b65a180fac8cb4158862fe99684e9abf8cc6c88571747453e"} Sep 11 14:19:27.375525 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.375432 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" event={"ID":"7dc26222-e5c6-4fac-8397-135f60eb8138","Type":"ContainerStarted","Data":"3230c824b767f74a8d1d5aface0f3d2e666c83771aea76f3fcef191ade404743"} Sep 11 14:19:27.531366 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.531284 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-64hd5"] Sep 11 14:19:27.534453 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.534436 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-64hd5" Sep 11 14:19:27.544945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.544923 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4f6q7\"" Sep 11 14:19:27.545126 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.545114 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 14:19:27.545179 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.545142 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 14:19:27.568333 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.568311 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-64hd5"] Sep 11 14:19:27.583940 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.583917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fhxk\" (UniqueName: \"kubernetes.io/projected/5e77779c-4abd-4816-a258-056b2e798400-kube-api-access-7fhxk\") pod \"cert-manager-webhook-6486ffd997-64hd5\" (UID: \"5e77779c-4abd-4816-a258-056b2e798400\") " pod="cert-manager/cert-manager-webhook-6486ffd997-64hd5" Sep 11 14:19:27.584225 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.583951 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e77779c-4abd-4816-a258-056b2e798400-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-64hd5\" (UID: \"5e77779c-4abd-4816-a258-056b2e798400\") " pod="cert-manager/cert-manager-webhook-6486ffd997-64hd5" Sep 11 14:19:27.684557 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.684526 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fhxk\" (UniqueName: \"kubernetes.io/projected/5e77779c-4abd-4816-a258-056b2e798400-kube-api-access-7fhxk\") pod \"cert-manager-webhook-6486ffd997-64hd5\" (UID: \"5e77779c-4abd-4816-a258-056b2e798400\") " pod="cert-manager/cert-manager-webhook-6486ffd997-64hd5" Sep 11 14:19:27.684557 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.684561 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e77779c-4abd-4816-a258-056b2e798400-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-64hd5\" (UID: \"5e77779c-4abd-4816-a258-056b2e798400\") " pod="cert-manager/cert-manager-webhook-6486ffd997-64hd5" Sep 11 14:19:27.704242 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.704213 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e77779c-4abd-4816-a258-056b2e798400-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-64hd5\" (UID: \"5e77779c-4abd-4816-a258-056b2e798400\") " pod="cert-manager/cert-manager-webhook-6486ffd997-64hd5" Sep 11 14:19:27.709574 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.709555 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fhxk\" (UniqueName: \"kubernetes.io/projected/5e77779c-4abd-4816-a258-056b2e798400-kube-api-access-7fhxk\") pod \"cert-manager-webhook-6486ffd997-64hd5\" (UID: \"5e77779c-4abd-4816-a258-056b2e798400\") " pod="cert-manager/cert-manager-webhook-6486ffd997-64hd5" Sep 11 14:19:27.850254 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:27.850221 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-64hd5" Sep 11 14:19:28.129699 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:28.129360 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-64hd5"] Sep 11 14:19:28.132032 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:19:28.132004 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e77779c_4abd_4816_a258_056b2e798400.slice/crio-dc28c8e4b7689eb1c82a7b7680ed5967381f515c6e0e9862d90474a79e2e7555 WatchSource:0}: Error finding container dc28c8e4b7689eb1c82a7b7680ed5967381f515c6e0e9862d90474a79e2e7555: Status 404 returned error can't find the container with id dc28c8e4b7689eb1c82a7b7680ed5967381f515c6e0e9862d90474a79e2e7555 Sep 11 14:19:28.380326 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:28.380255 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-64hd5" event={"ID":"5e77779c-4abd-4816-a258-056b2e798400","Type":"ContainerStarted","Data":"dc28c8e4b7689eb1c82a7b7680ed5967381f515c6e0e9862d90474a79e2e7555"} Sep 11 14:19:32.401259 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:32.401213 2573 generic.go:356] "Generic (PLEG): container finished" podID="7dc26222-e5c6-4fac-8397-135f60eb8138" containerID="a874894aa3ba18e8060ed2a999c6e4901a40517b0eb78855d0edcd8f1d564b81" exitCode=0 Sep 11 14:19:32.401675 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:32.401311 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" event={"ID":"7dc26222-e5c6-4fac-8397-135f60eb8138","Type":"ContainerDied","Data":"a874894aa3ba18e8060ed2a999c6e4901a40517b0eb78855d0edcd8f1d564b81"} Sep 11 14:19:32.403228 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:32.403205 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-64hd5" event={"ID":"5e77779c-4abd-4816-a258-056b2e798400","Type":"ContainerStarted","Data":"1f0086adff27b53d1b70399a3c04eec2092c13b9c20245e71a659f536538dfb5"} Sep 11 14:19:32.403333 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:32.403269 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-64hd5" Sep 11 14:19:32.606298 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:32.606250 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-64hd5" podStartSLOduration=2.10418857 podStartE2EDuration="5.606233327s" totalImagesPullingTime="3.502044757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:32.605545065 +0000 UTC m=+284.942513255" watchObservedRunningTime="2026-09-11 14:19:32.606233327 +0000 UTC m=+284.943201519" Sep 11 14:19:33.409430 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:33.409397 2573 generic.go:356] "Generic (PLEG): container finished" podID="7dc26222-e5c6-4fac-8397-135f60eb8138" containerID="d85245ec843b0fe9e5f3e33b0d79b726584144a95af131adf662671cda4a7c66" exitCode=0 Sep 11 14:19:33.409905 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:33.409491 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" event={"ID":"7dc26222-e5c6-4fac-8397-135f60eb8138","Type":"ContainerDied","Data":"d85245ec843b0fe9e5f3e33b0d79b726584144a95af131adf662671cda4a7c66"} Sep 11 14:19:34.535250 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:34.535225 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" Sep 11 14:19:34.641446 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:34.641417 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7dc26222-e5c6-4fac-8397-135f60eb8138-bundle\") pod \"7dc26222-e5c6-4fac-8397-135f60eb8138\" (UID: \"7dc26222-e5c6-4fac-8397-135f60eb8138\") " Sep 11 14:19:34.641592 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:34.641523 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xhsx\" (UniqueName: \"kubernetes.io/projected/7dc26222-e5c6-4fac-8397-135f60eb8138-kube-api-access-8xhsx\") pod \"7dc26222-e5c6-4fac-8397-135f60eb8138\" (UID: \"7dc26222-e5c6-4fac-8397-135f60eb8138\") " Sep 11 14:19:34.641592 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:34.641578 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7dc26222-e5c6-4fac-8397-135f60eb8138-util\") pod \"7dc26222-e5c6-4fac-8397-135f60eb8138\" (UID: \"7dc26222-e5c6-4fac-8397-135f60eb8138\") " Sep 11 14:19:34.641817 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:34.641800 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc26222-e5c6-4fac-8397-135f60eb8138-bundle" (OuterVolumeSpecName: "bundle") pod "7dc26222-e5c6-4fac-8397-135f60eb8138" (UID: "7dc26222-e5c6-4fac-8397-135f60eb8138"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:34.643564 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:34.643537 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc26222-e5c6-4fac-8397-135f60eb8138-kube-api-access-8xhsx" (OuterVolumeSpecName: "kube-api-access-8xhsx") pod "7dc26222-e5c6-4fac-8397-135f60eb8138" (UID: "7dc26222-e5c6-4fac-8397-135f60eb8138"). InnerVolumeSpecName "kube-api-access-8xhsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:34.645807 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:34.645787 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc26222-e5c6-4fac-8397-135f60eb8138-util" (OuterVolumeSpecName: "util") pod "7dc26222-e5c6-4fac-8397-135f60eb8138" (UID: "7dc26222-e5c6-4fac-8397-135f60eb8138"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:34.742563 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:34.742497 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7dc26222-e5c6-4fac-8397-135f60eb8138-util\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:19:34.742563 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:34.742521 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7dc26222-e5c6-4fac-8397-135f60eb8138-bundle\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:19:34.742563 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:34.742532 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xhsx\" (UniqueName: \"kubernetes.io/projected/7dc26222-e5c6-4fac-8397-135f60eb8138-kube-api-access-8xhsx\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:19:35.420120 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:35.420087 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" event={"ID":"7dc26222-e5c6-4fac-8397-135f60eb8138","Type":"ContainerDied","Data":"3230c824b767f74a8d1d5aface0f3d2e666c83771aea76f3fcef191ade404743"} Sep 11 14:19:35.420282 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:35.420128 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7h9tw" Sep 11 14:19:35.420282 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:35.420133 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3230c824b767f74a8d1d5aface0f3d2e666c83771aea76f3fcef191ade404743" Sep 11 14:19:37.048010 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.047977 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rk5tm"] Sep 11 14:19:37.048360 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.048289 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dc26222-e5c6-4fac-8397-135f60eb8138" containerName="pull" Sep 11 14:19:37.048360 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.048300 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dc26222-e5c6-4fac-8397-135f60eb8138" containerName="pull" Sep 11 14:19:37.048360 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.048312 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dc26222-e5c6-4fac-8397-135f60eb8138" containerName="extract" Sep 11 14:19:37.048360 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.048318 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dc26222-e5c6-4fac-8397-135f60eb8138" containerName="extract" Sep 11 14:19:37.048360 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.048331 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dc26222-e5c6-4fac-8397-135f60eb8138" containerName="util" Sep 11 14:19:37.048360 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.048337 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dc26222-e5c6-4fac-8397-135f60eb8138" containerName="util" Sep 11 14:19:37.048550 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.048416 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dc26222-e5c6-4fac-8397-135f60eb8138" containerName="extract" Sep 11 14:19:37.052392 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.052375 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rk5tm" Sep 11 14:19:37.060032 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.060015 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-tnfrm\"" Sep 11 14:19:37.078821 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.078799 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rk5tm"] Sep 11 14:19:37.163080 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.163053 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/068cc468-bcfe-42fd-aa7a-d34db9f46e4d-bound-sa-token\") pod \"cert-manager-55b4d954d5-rk5tm\" (UID: \"068cc468-bcfe-42fd-aa7a-d34db9f46e4d\") " pod="cert-manager/cert-manager-55b4d954d5-rk5tm" Sep 11 14:19:37.163209 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.163100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pfbt\" (UniqueName: \"kubernetes.io/projected/068cc468-bcfe-42fd-aa7a-d34db9f46e4d-kube-api-access-4pfbt\") pod \"cert-manager-55b4d954d5-rk5tm\" (UID: \"068cc468-bcfe-42fd-aa7a-d34db9f46e4d\") " pod="cert-manager/cert-manager-55b4d954d5-rk5tm" Sep 11 14:19:37.264004 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.263967 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/068cc468-bcfe-42fd-aa7a-d34db9f46e4d-bound-sa-token\") pod \"cert-manager-55b4d954d5-rk5tm\" (UID: \"068cc468-bcfe-42fd-aa7a-d34db9f46e4d\") " pod="cert-manager/cert-manager-55b4d954d5-rk5tm" Sep 11 14:19:37.264201 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.264025 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pfbt\" (UniqueName: \"kubernetes.io/projected/068cc468-bcfe-42fd-aa7a-d34db9f46e4d-kube-api-access-4pfbt\") pod \"cert-manager-55b4d954d5-rk5tm\" (UID: \"068cc468-bcfe-42fd-aa7a-d34db9f46e4d\") " pod="cert-manager/cert-manager-55b4d954d5-rk5tm" Sep 11 14:19:37.282014 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.281982 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/068cc468-bcfe-42fd-aa7a-d34db9f46e4d-bound-sa-token\") pod \"cert-manager-55b4d954d5-rk5tm\" (UID: \"068cc468-bcfe-42fd-aa7a-d34db9f46e4d\") " pod="cert-manager/cert-manager-55b4d954d5-rk5tm" Sep 11 14:19:37.284041 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.284014 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pfbt\" (UniqueName: \"kubernetes.io/projected/068cc468-bcfe-42fd-aa7a-d34db9f46e4d-kube-api-access-4pfbt\") pod \"cert-manager-55b4d954d5-rk5tm\" (UID: \"068cc468-bcfe-42fd-aa7a-d34db9f46e4d\") " pod="cert-manager/cert-manager-55b4d954d5-rk5tm" Sep 11 14:19:37.361011 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.360976 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rk5tm" Sep 11 14:19:37.648304 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:37.648196 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rk5tm"] Sep 11 14:19:37.653339 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:19:37.653308 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod068cc468_bcfe_42fd_aa7a_d34db9f46e4d.slice/crio-13f9ffe7d8ad93f68df779be022eb85b3964506a731c974ff84fd498e8c6a163 WatchSource:0}: Error finding container 13f9ffe7d8ad93f68df779be022eb85b3964506a731c974ff84fd498e8c6a163: Status 404 returned error can't find the container with id 13f9ffe7d8ad93f68df779be022eb85b3964506a731c974ff84fd498e8c6a163 Sep 11 14:19:38.411332 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:38.411303 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-64hd5" Sep 11 14:19:38.435234 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:38.435205 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rk5tm" event={"ID":"068cc468-bcfe-42fd-aa7a-d34db9f46e4d","Type":"ContainerStarted","Data":"93e894dbb014018883cfb5a7d803869c1a53390fdea1aac97937cace4efef5b4"} Sep 11 14:19:38.435340 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:38.435240 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rk5tm" event={"ID":"068cc468-bcfe-42fd-aa7a-d34db9f46e4d","Type":"ContainerStarted","Data":"13f9ffe7d8ad93f68df779be022eb85b3964506a731c974ff84fd498e8c6a163"} Sep 11 14:19:38.492639 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:38.492592 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-rk5tm" podStartSLOduration=1.492576092 podStartE2EDuration="1.492576092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:38.49193958 +0000 UTC m=+290.828907770" watchObservedRunningTime="2026-09-11 14:19:38.492576092 +0000 UTC m=+290.829544283" Sep 11 14:19:46.403548 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:46.403511 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj"] Sep 11 14:19:46.406012 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:46.405993 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" Sep 11 14:19:46.408671 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:46.408649 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:19:46.409668 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:46.409637 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:19:46.409668 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:46.409660 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4brs6\"" Sep 11 14:19:46.418055 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:46.418031 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj"] Sep 11 14:19:46.537815 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:46.537778 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkc2j\" (UniqueName: \"kubernetes.io/projected/cc783e93-f77f-4d6f-b396-d8f870861c88-kube-api-access-jkc2j\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj\" (UID: \"cc783e93-f77f-4d6f-b396-d8f870861c88\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" Sep 11 14:19:46.537815 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:46.537819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc783e93-f77f-4d6f-b396-d8f870861c88-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj\" (UID: \"cc783e93-f77f-4d6f-b396-d8f870861c88\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" Sep 11 14:19:46.538049 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:46.537895 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc783e93-f77f-4d6f-b396-d8f870861c88-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj\" (UID: \"cc783e93-f77f-4d6f-b396-d8f870861c88\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" Sep 11 14:19:46.638555 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:46.638528 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkc2j\" (UniqueName: \"kubernetes.io/projected/cc783e93-f77f-4d6f-b396-d8f870861c88-kube-api-access-jkc2j\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj\" (UID: \"cc783e93-f77f-4d6f-b396-d8f870861c88\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" Sep 11 14:19:46.638711 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:46.638561 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc783e93-f77f-4d6f-b396-d8f870861c88-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj\" (UID: \"cc783e93-f77f-4d6f-b396-d8f870861c88\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" Sep 11 14:19:46.638711 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:46.638580 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc783e93-f77f-4d6f-b396-d8f870861c88-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj\" (UID: \"cc783e93-f77f-4d6f-b396-d8f870861c88\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" Sep 11 14:19:46.639027 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:46.639007 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc783e93-f77f-4d6f-b396-d8f870861c88-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj\" (UID: \"cc783e93-f77f-4d6f-b396-d8f870861c88\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" Sep 11 14:19:46.639073 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:46.639017 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc783e93-f77f-4d6f-b396-d8f870861c88-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj\" (UID: \"cc783e93-f77f-4d6f-b396-d8f870861c88\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" Sep 11 14:19:46.650683 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:46.650662 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkc2j\" (UniqueName: \"kubernetes.io/projected/cc783e93-f77f-4d6f-b396-d8f870861c88-kube-api-access-jkc2j\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj\" (UID: \"cc783e93-f77f-4d6f-b396-d8f870861c88\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" Sep 11 14:19:46.715419 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:46.715365 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" Sep 11 14:19:47.001043 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:47.001014 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj"] Sep 11 14:19:47.002290 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:19:47.002265 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc783e93_f77f_4d6f_b396_d8f870861c88.slice/crio-5e2bf9c6b6b67badeb608c7f1bb3d90883995a4aa81e4d87a15726da410df242 WatchSource:0}: Error finding container 5e2bf9c6b6b67badeb608c7f1bb3d90883995a4aa81e4d87a15726da410df242: Status 404 returned error can't find the container with id 5e2bf9c6b6b67badeb608c7f1bb3d90883995a4aa81e4d87a15726da410df242 Sep 11 14:19:47.475949 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:47.475912 2573 generic.go:356] "Generic (PLEG): container finished" podID="cc783e93-f77f-4d6f-b396-d8f870861c88" containerID="b650fe8f2fef9297fd55e08ed6c38a346fa567d69c4d0c69bd6ebe7b92a36ac3" exitCode=0 Sep 11 14:19:47.476342 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:47.475947 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" event={"ID":"cc783e93-f77f-4d6f-b396-d8f870861c88","Type":"ContainerDied","Data":"b650fe8f2fef9297fd55e08ed6c38a346fa567d69c4d0c69bd6ebe7b92a36ac3"} Sep 11 14:19:47.476342 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:47.475993 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" event={"ID":"cc783e93-f77f-4d6f-b396-d8f870861c88","Type":"ContainerStarted","Data":"5e2bf9c6b6b67badeb608c7f1bb3d90883995a4aa81e4d87a15726da410df242"} Sep 11 14:19:48.113256 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:48.113233 2573 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 14:19:48.482571 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:48.482536 2573 generic.go:356] "Generic (PLEG): container finished" podID="cc783e93-f77f-4d6f-b396-d8f870861c88" containerID="9564046f9340d2597cead18d75f449384d6d8150d98632bef918a40a80533ea6" exitCode=0 Sep 11 14:19:48.484033 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:48.482624 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" event={"ID":"cc783e93-f77f-4d6f-b396-d8f870861c88","Type":"ContainerDied","Data":"9564046f9340d2597cead18d75f449384d6d8150d98632bef918a40a80533ea6"} Sep 11 14:19:49.489370 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:49.489335 2573 generic.go:356] "Generic (PLEG): container finished" podID="cc783e93-f77f-4d6f-b396-d8f870861c88" containerID="185e35ac6919ed56a03af182449edd38b2cdc628de0a24df274e82c67ac79a31" exitCode=0 Sep 11 14:19:49.489761 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:49.489426 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" event={"ID":"cc783e93-f77f-4d6f-b396-d8f870861c88","Type":"ContainerDied","Data":"185e35ac6919ed56a03af182449edd38b2cdc628de0a24df274e82c67ac79a31"} Sep 11 14:19:50.615633 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:50.615610 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" Sep 11 14:19:50.772933 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:50.772852 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkc2j\" (UniqueName: \"kubernetes.io/projected/cc783e93-f77f-4d6f-b396-d8f870861c88-kube-api-access-jkc2j\") pod \"cc783e93-f77f-4d6f-b396-d8f870861c88\" (UID: \"cc783e93-f77f-4d6f-b396-d8f870861c88\") " Sep 11 14:19:50.772933 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:50.772912 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc783e93-f77f-4d6f-b396-d8f870861c88-bundle\") pod \"cc783e93-f77f-4d6f-b396-d8f870861c88\" (UID: \"cc783e93-f77f-4d6f-b396-d8f870861c88\") " Sep 11 14:19:50.773104 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:50.773015 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc783e93-f77f-4d6f-b396-d8f870861c88-util\") pod \"cc783e93-f77f-4d6f-b396-d8f870861c88\" (UID: \"cc783e93-f77f-4d6f-b396-d8f870861c88\") " Sep 11 14:19:50.773598 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:50.773572 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc783e93-f77f-4d6f-b396-d8f870861c88-bundle" (OuterVolumeSpecName: "bundle") pod "cc783e93-f77f-4d6f-b396-d8f870861c88" (UID: "cc783e93-f77f-4d6f-b396-d8f870861c88"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:50.774893 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:50.774864 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc783e93-f77f-4d6f-b396-d8f870861c88-kube-api-access-jkc2j" (OuterVolumeSpecName: "kube-api-access-jkc2j") pod "cc783e93-f77f-4d6f-b396-d8f870861c88" (UID: "cc783e93-f77f-4d6f-b396-d8f870861c88"). InnerVolumeSpecName "kube-api-access-jkc2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:50.778155 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:50.778136 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc783e93-f77f-4d6f-b396-d8f870861c88-util" (OuterVolumeSpecName: "util") pod "cc783e93-f77f-4d6f-b396-d8f870861c88" (UID: "cc783e93-f77f-4d6f-b396-d8f870861c88"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:50.874478 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:50.874449 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkc2j\" (UniqueName: \"kubernetes.io/projected/cc783e93-f77f-4d6f-b396-d8f870861c88-kube-api-access-jkc2j\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:19:50.874478 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:50.874472 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc783e93-f77f-4d6f-b396-d8f870861c88-bundle\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:19:50.874478 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:50.874481 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc783e93-f77f-4d6f-b396-d8f870861c88-util\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:19:51.500606 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:51.500576 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" Sep 11 14:19:51.500802 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:51.500607 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5dgsj" event={"ID":"cc783e93-f77f-4d6f-b396-d8f870861c88","Type":"ContainerDied","Data":"5e2bf9c6b6b67badeb608c7f1bb3d90883995a4aa81e4d87a15726da410df242"} Sep 11 14:19:51.500802 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:51.500654 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e2bf9c6b6b67badeb608c7f1bb3d90883995a4aa81e4d87a15726da410df242" Sep 11 14:19:58.601329 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.601294 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9"] Sep 11 14:19:58.601707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.601605 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc783e93-f77f-4d6f-b396-d8f870861c88" containerName="util" Sep 11 14:19:58.601707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.601616 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc783e93-f77f-4d6f-b396-d8f870861c88" containerName="util" Sep 11 14:19:58.601707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.601625 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc783e93-f77f-4d6f-b396-d8f870861c88" containerName="pull" Sep 11 14:19:58.601707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.601630 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc783e93-f77f-4d6f-b396-d8f870861c88" containerName="pull" Sep 11 14:19:58.601707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.601647 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc783e93-f77f-4d6f-b396-d8f870861c88" containerName="extract" Sep 11 14:19:58.601707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.601652 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc783e93-f77f-4d6f-b396-d8f870861c88" containerName="extract" Sep 11 14:19:58.601924 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.601714 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc783e93-f77f-4d6f-b396-d8f870861c88" containerName="extract" Sep 11 14:19:58.603886 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.603869 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" Sep 11 14:19:58.608174 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.608134 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4brs6\"" Sep 11 14:19:58.608484 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.608464 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:19:58.612341 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.612117 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:19:58.641529 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.641503 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9"] Sep 11 14:19:58.736669 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.736642 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f135db8-3099-49bd-82f7-fbc85d343de7-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9\" (UID: \"9f135db8-3099-49bd-82f7-fbc85d343de7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" Sep 11 14:19:58.736812 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.736676 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f135db8-3099-49bd-82f7-fbc85d343de7-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9\" (UID: \"9f135db8-3099-49bd-82f7-fbc85d343de7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" Sep 11 14:19:58.736812 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.736701 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9mq7\" (UniqueName: \"kubernetes.io/projected/9f135db8-3099-49bd-82f7-fbc85d343de7-kube-api-access-h9mq7\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9\" (UID: \"9f135db8-3099-49bd-82f7-fbc85d343de7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" Sep 11 14:19:58.837102 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.837072 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f135db8-3099-49bd-82f7-fbc85d343de7-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9\" (UID: \"9f135db8-3099-49bd-82f7-fbc85d343de7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" Sep 11 14:19:58.837256 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.837104 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f135db8-3099-49bd-82f7-fbc85d343de7-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9\" (UID: \"9f135db8-3099-49bd-82f7-fbc85d343de7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" Sep 11 14:19:58.837256 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.837129 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9mq7\" (UniqueName: \"kubernetes.io/projected/9f135db8-3099-49bd-82f7-fbc85d343de7-kube-api-access-h9mq7\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9\" (UID: \"9f135db8-3099-49bd-82f7-fbc85d343de7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" Sep 11 14:19:58.837431 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.837408 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f135db8-3099-49bd-82f7-fbc85d343de7-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9\" (UID: \"9f135db8-3099-49bd-82f7-fbc85d343de7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" Sep 11 14:19:58.837508 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.837477 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f135db8-3099-49bd-82f7-fbc85d343de7-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9\" (UID: \"9f135db8-3099-49bd-82f7-fbc85d343de7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" Sep 11 14:19:58.873443 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.873389 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9mq7\" (UniqueName: \"kubernetes.io/projected/9f135db8-3099-49bd-82f7-fbc85d343de7-kube-api-access-h9mq7\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9\" (UID: \"9f135db8-3099-49bd-82f7-fbc85d343de7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" Sep 11 14:19:58.917179 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:58.917153 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" Sep 11 14:19:59.201187 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:59.201160 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9"] Sep 11 14:19:59.203766 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:19:59.203743 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f135db8_3099_49bd_82f7_fbc85d343de7.slice/crio-c945305269392919918c1aac67d73698928636959f7f2701e87ec9407a2430b5 WatchSource:0}: Error finding container c945305269392919918c1aac67d73698928636959f7f2701e87ec9407a2430b5: Status 404 returned error can't find the container with id c945305269392919918c1aac67d73698928636959f7f2701e87ec9407a2430b5 Sep 11 14:19:59.537951 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:59.537868 2573 generic.go:356] "Generic (PLEG): container finished" podID="9f135db8-3099-49bd-82f7-fbc85d343de7" containerID="15e5854cbb6eb74000be1a19cd26cc3bbab4c5f36e8a8381805b87a0bb8cdf0f" exitCode=0 Sep 11 14:19:59.538111 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:59.537991 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" event={"ID":"9f135db8-3099-49bd-82f7-fbc85d343de7","Type":"ContainerDied","Data":"15e5854cbb6eb74000be1a19cd26cc3bbab4c5f36e8a8381805b87a0bb8cdf0f"} Sep 11 14:19:59.538111 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:59.538035 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" event={"ID":"9f135db8-3099-49bd-82f7-fbc85d343de7","Type":"ContainerStarted","Data":"c945305269392919918c1aac67d73698928636959f7f2701e87ec9407a2430b5"} Sep 11 14:19:59.538506 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:19:59.538491 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:20:00.544944 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:00.544864 2573 generic.go:356] "Generic (PLEG): container finished" podID="9f135db8-3099-49bd-82f7-fbc85d343de7" containerID="b89c6fd5731c85f85ddd8a82b24040dde1c9b35e71426036ed3cf56cb4d06653" exitCode=0 Sep 11 14:20:00.544944 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:00.544923 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" event={"ID":"9f135db8-3099-49bd-82f7-fbc85d343de7","Type":"ContainerDied","Data":"b89c6fd5731c85f85ddd8a82b24040dde1c9b35e71426036ed3cf56cb4d06653"} Sep 11 14:20:01.552411 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:01.552374 2573 generic.go:356] "Generic (PLEG): container finished" podID="9f135db8-3099-49bd-82f7-fbc85d343de7" containerID="3ab9070425c3d258a53407032e2cb939599cd95e03f8b8b0b040fe1fbfab0c92" exitCode=0 Sep 11 14:20:01.552789 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:01.552449 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" event={"ID":"9f135db8-3099-49bd-82f7-fbc85d343de7","Type":"ContainerDied","Data":"3ab9070425c3d258a53407032e2cb939599cd95e03f8b8b0b040fe1fbfab0c92"} Sep 11 14:20:01.793239 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:01.793198 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5"] Sep 11 14:20:01.795648 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:01.795626 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:01.806915 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:01.806865 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 14:20:01.807245 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:01.807227 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 14:20:01.807558 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:01.807542 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 14:20:01.808026 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:01.808011 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 14:20:01.825475 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:01.825458 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rpcgr\"" Sep 11 14:20:01.841577 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:01.841555 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5"] Sep 11 14:20:01.962411 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:01.962363 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5358a5de-1c21-4cfc-a220-9a45c8e1d934-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74798b5f56-mhrv5\" (UID: \"5358a5de-1c21-4cfc-a220-9a45c8e1d934\") " pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:01.962598 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:01.962419 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpz5\" (UniqueName: \"kubernetes.io/projected/5358a5de-1c21-4cfc-a220-9a45c8e1d934-kube-api-access-8lpz5\") pod \"opendatahub-operator-controller-manager-74798b5f56-mhrv5\" (UID: \"5358a5de-1c21-4cfc-a220-9a45c8e1d934\") " pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:01.962598 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:01.962485 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5358a5de-1c21-4cfc-a220-9a45c8e1d934-webhook-cert\") pod \"opendatahub-operator-controller-manager-74798b5f56-mhrv5\" (UID: \"5358a5de-1c21-4cfc-a220-9a45c8e1d934\") " pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:01.962598 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:01.962524 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5358a5de-1c21-4cfc-a220-9a45c8e1d934-manifests\") pod \"opendatahub-operator-controller-manager-74798b5f56-mhrv5\" (UID: \"5358a5de-1c21-4cfc-a220-9a45c8e1d934\") " pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:01.962598 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:01.962579 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5358a5de-1c21-4cfc-a220-9a45c8e1d934-tmp\") pod \"opendatahub-operator-controller-manager-74798b5f56-mhrv5\" (UID: \"5358a5de-1c21-4cfc-a220-9a45c8e1d934\") " pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:02.063588 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.063506 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5358a5de-1c21-4cfc-a220-9a45c8e1d934-webhook-cert\") pod \"opendatahub-operator-controller-manager-74798b5f56-mhrv5\" (UID: \"5358a5de-1c21-4cfc-a220-9a45c8e1d934\") " pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:02.063588 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.063546 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5358a5de-1c21-4cfc-a220-9a45c8e1d934-manifests\") pod \"opendatahub-operator-controller-manager-74798b5f56-mhrv5\" (UID: \"5358a5de-1c21-4cfc-a220-9a45c8e1d934\") " pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:02.063588 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.063579 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5358a5de-1c21-4cfc-a220-9a45c8e1d934-tmp\") pod \"opendatahub-operator-controller-manager-74798b5f56-mhrv5\" (UID: \"5358a5de-1c21-4cfc-a220-9a45c8e1d934\") " pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:02.063874 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.063614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5358a5de-1c21-4cfc-a220-9a45c8e1d934-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74798b5f56-mhrv5\" (UID: \"5358a5de-1c21-4cfc-a220-9a45c8e1d934\") " pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:02.063874 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.063632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpz5\" (UniqueName: \"kubernetes.io/projected/5358a5de-1c21-4cfc-a220-9a45c8e1d934-kube-api-access-8lpz5\") pod \"opendatahub-operator-controller-manager-74798b5f56-mhrv5\" (UID: \"5358a5de-1c21-4cfc-a220-9a45c8e1d934\") " pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:02.064039 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.064014 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5358a5de-1c21-4cfc-a220-9a45c8e1d934-manifests\") pod \"opendatahub-operator-controller-manager-74798b5f56-mhrv5\" (UID: \"5358a5de-1c21-4cfc-a220-9a45c8e1d934\") " pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:02.066000 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.065968 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5358a5de-1c21-4cfc-a220-9a45c8e1d934-tmp\") pod \"opendatahub-operator-controller-manager-74798b5f56-mhrv5\" (UID: \"5358a5de-1c21-4cfc-a220-9a45c8e1d934\") " pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:02.066124 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.066104 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5358a5de-1c21-4cfc-a220-9a45c8e1d934-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74798b5f56-mhrv5\" (UID: \"5358a5de-1c21-4cfc-a220-9a45c8e1d934\") " pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:02.066182 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.066106 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5358a5de-1c21-4cfc-a220-9a45c8e1d934-webhook-cert\") pod \"opendatahub-operator-controller-manager-74798b5f56-mhrv5\" (UID: \"5358a5de-1c21-4cfc-a220-9a45c8e1d934\") " pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:02.081208 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.081179 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpz5\" (UniqueName: \"kubernetes.io/projected/5358a5de-1c21-4cfc-a220-9a45c8e1d934-kube-api-access-8lpz5\") pod \"opendatahub-operator-controller-manager-74798b5f56-mhrv5\" (UID: \"5358a5de-1c21-4cfc-a220-9a45c8e1d934\") " pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:02.105961 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.105932 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:02.429544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.424233 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5"] Sep 11 14:20:02.560700 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.560668 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" event={"ID":"5358a5de-1c21-4cfc-a220-9a45c8e1d934","Type":"ContainerStarted","Data":"be4f0e0b8facc399714e57ef8f14c4f386798b1c978fed069c4d15b756594a76"} Sep 11 14:20:02.672979 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.672956 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" Sep 11 14:20:02.768683 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.768612 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f135db8-3099-49bd-82f7-fbc85d343de7-util\") pod \"9f135db8-3099-49bd-82f7-fbc85d343de7\" (UID: \"9f135db8-3099-49bd-82f7-fbc85d343de7\") " Sep 11 14:20:02.768855 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.768689 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9mq7\" (UniqueName: \"kubernetes.io/projected/9f135db8-3099-49bd-82f7-fbc85d343de7-kube-api-access-h9mq7\") pod \"9f135db8-3099-49bd-82f7-fbc85d343de7\" (UID: \"9f135db8-3099-49bd-82f7-fbc85d343de7\") " Sep 11 14:20:02.768855 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.768708 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f135db8-3099-49bd-82f7-fbc85d343de7-bundle\") pod \"9f135db8-3099-49bd-82f7-fbc85d343de7\" (UID: \"9f135db8-3099-49bd-82f7-fbc85d343de7\") " Sep 11 14:20:02.769481 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.769448 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f135db8-3099-49bd-82f7-fbc85d343de7-bundle" (OuterVolumeSpecName: "bundle") pod "9f135db8-3099-49bd-82f7-fbc85d343de7" (UID: "9f135db8-3099-49bd-82f7-fbc85d343de7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:20:02.770733 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.770709 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f135db8-3099-49bd-82f7-fbc85d343de7-kube-api-access-h9mq7" (OuterVolumeSpecName: "kube-api-access-h9mq7") pod "9f135db8-3099-49bd-82f7-fbc85d343de7" (UID: "9f135db8-3099-49bd-82f7-fbc85d343de7"). InnerVolumeSpecName "kube-api-access-h9mq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:20:02.774523 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.774502 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f135db8-3099-49bd-82f7-fbc85d343de7-util" (OuterVolumeSpecName: "util") pod "9f135db8-3099-49bd-82f7-fbc85d343de7" (UID: "9f135db8-3099-49bd-82f7-fbc85d343de7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:20:02.869650 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.869625 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9mq7\" (UniqueName: \"kubernetes.io/projected/9f135db8-3099-49bd-82f7-fbc85d343de7-kube-api-access-h9mq7\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:20:02.869650 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.869647 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f135db8-3099-49bd-82f7-fbc85d343de7-bundle\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:20:02.869793 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:02.869657 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f135db8-3099-49bd-82f7-fbc85d343de7-util\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:20:03.570231 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:03.570194 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" event={"ID":"9f135db8-3099-49bd-82f7-fbc85d343de7","Type":"ContainerDied","Data":"c945305269392919918c1aac67d73698928636959f7f2701e87ec9407a2430b5"} Sep 11 14:20:03.570231 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:03.570219 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qkdr9" Sep 11 14:20:03.570734 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:03.570253 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c945305269392919918c1aac67d73698928636959f7f2701e87ec9407a2430b5" Sep 11 14:20:06.586891 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:06.586853 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" event={"ID":"5358a5de-1c21-4cfc-a220-9a45c8e1d934","Type":"ContainerStarted","Data":"28188ce3fe821ba0717eb6aab49628dae42bc5ad6b9505d6657f14e976816b21"} Sep 11 14:20:07.593030 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:07.592995 2573 generic.go:356] "Generic (PLEG): container finished" podID="5358a5de-1c21-4cfc-a220-9a45c8e1d934" containerID="28188ce3fe821ba0717eb6aab49628dae42bc5ad6b9505d6657f14e976816b21" exitCode=0 Sep 11 14:20:07.593438 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:07.593084 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" event={"ID":"5358a5de-1c21-4cfc-a220-9a45c8e1d934","Type":"ContainerDied","Data":"28188ce3fe821ba0717eb6aab49628dae42bc5ad6b9505d6657f14e976816b21"} Sep 11 14:20:08.599721 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:08.599685 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" event={"ID":"5358a5de-1c21-4cfc-a220-9a45c8e1d934","Type":"ContainerStarted","Data":"8b1ee7bec75cbd6b98bf4e176aa103052db191b8d29c72cea68918035381cc2d"} Sep 11 14:20:08.600153 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:08.599814 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:08.643727 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:08.643678 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" podStartSLOduration=3.07676711 podStartE2EDuration="7.643663653s" totalImagesPullingTime="3.485393794s" totalInitContainerRuntime="1.081502749s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:08.639993377 +0000 UTC m=+320.976961578" watchObservedRunningTime="2026-09-11 14:20:08.643663653 +0000 UTC m=+320.980631844" Sep 11 14:20:19.606241 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:19.606204 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-74798b5f56-mhrv5" Sep 11 14:20:28.298511 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.298477 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-f974d944c-jkqks"] Sep 11 14:20:28.298879 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.298794 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f135db8-3099-49bd-82f7-fbc85d343de7" containerName="extract" Sep 11 14:20:28.298879 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.298806 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f135db8-3099-49bd-82f7-fbc85d343de7" containerName="extract" Sep 11 14:20:28.298879 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.298824 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f135db8-3099-49bd-82f7-fbc85d343de7" containerName="pull" Sep 11 14:20:28.298879 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.298842 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f135db8-3099-49bd-82f7-fbc85d343de7" containerName="pull" Sep 11 14:20:28.298879 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.298856 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f135db8-3099-49bd-82f7-fbc85d343de7" containerName="util" Sep 11 14:20:28.298879 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.298862 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f135db8-3099-49bd-82f7-fbc85d343de7" containerName="util" Sep 11 14:20:28.299066 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.298916 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f135db8-3099-49bd-82f7-fbc85d343de7" containerName="extract" Sep 11 14:20:28.307728 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.307704 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f974d944c-jkqks" Sep 11 14:20:28.313481 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.313456 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 14:20:28.314803 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.314771 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 11 14:20:28.314803 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.314790 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 14:20:28.314955 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.314874 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 11 14:20:28.323262 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.323244 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fw7gh\"" Sep 11 14:20:28.326173 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.326152 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f974d944c-jkqks"] Sep 11 14:20:28.346184 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.346158 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71beef92-b67b-4ca2-82ec-803a0fb0f65c-tmp\") pod \"kube-auth-proxy-f974d944c-jkqks\" (UID: \"71beef92-b67b-4ca2-82ec-803a0fb0f65c\") " pod="openshift-ingress/kube-auth-proxy-f974d944c-jkqks" Sep 11 14:20:28.346265 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.346199 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71beef92-b67b-4ca2-82ec-803a0fb0f65c-tls-certs\") pod \"kube-auth-proxy-f974d944c-jkqks\" (UID: \"71beef92-b67b-4ca2-82ec-803a0fb0f65c\") " pod="openshift-ingress/kube-auth-proxy-f974d944c-jkqks" Sep 11 14:20:28.346265 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.346229 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvsvc\" (UniqueName: \"kubernetes.io/projected/71beef92-b67b-4ca2-82ec-803a0fb0f65c-kube-api-access-vvsvc\") pod \"kube-auth-proxy-f974d944c-jkqks\" (UID: \"71beef92-b67b-4ca2-82ec-803a0fb0f65c\") " pod="openshift-ingress/kube-auth-proxy-f974d944c-jkqks" Sep 11 14:20:28.446799 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.446777 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71beef92-b67b-4ca2-82ec-803a0fb0f65c-tmp\") pod \"kube-auth-proxy-f974d944c-jkqks\" (UID: \"71beef92-b67b-4ca2-82ec-803a0fb0f65c\") " pod="openshift-ingress/kube-auth-proxy-f974d944c-jkqks" Sep 11 14:20:28.446958 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.446828 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71beef92-b67b-4ca2-82ec-803a0fb0f65c-tls-certs\") pod \"kube-auth-proxy-f974d944c-jkqks\" (UID: \"71beef92-b67b-4ca2-82ec-803a0fb0f65c\") " pod="openshift-ingress/kube-auth-proxy-f974d944c-jkqks" Sep 11 14:20:28.446958 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.446909 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvsvc\" (UniqueName: \"kubernetes.io/projected/71beef92-b67b-4ca2-82ec-803a0fb0f65c-kube-api-access-vvsvc\") pod \"kube-auth-proxy-f974d944c-jkqks\" (UID: \"71beef92-b67b-4ca2-82ec-803a0fb0f65c\") " pod="openshift-ingress/kube-auth-proxy-f974d944c-jkqks" Sep 11 14:20:28.453281 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.450125 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71beef92-b67b-4ca2-82ec-803a0fb0f65c-tls-certs\") pod \"kube-auth-proxy-f974d944c-jkqks\" (UID: \"71beef92-b67b-4ca2-82ec-803a0fb0f65c\") " pod="openshift-ingress/kube-auth-proxy-f974d944c-jkqks" Sep 11 14:20:28.453886 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.453857 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71beef92-b67b-4ca2-82ec-803a0fb0f65c-tmp\") pod \"kube-auth-proxy-f974d944c-jkqks\" (UID: \"71beef92-b67b-4ca2-82ec-803a0fb0f65c\") " pod="openshift-ingress/kube-auth-proxy-f974d944c-jkqks" Sep 11 14:20:28.505647 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.505614 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvsvc\" (UniqueName: \"kubernetes.io/projected/71beef92-b67b-4ca2-82ec-803a0fb0f65c-kube-api-access-vvsvc\") pod \"kube-auth-proxy-f974d944c-jkqks\" (UID: \"71beef92-b67b-4ca2-82ec-803a0fb0f65c\") " pod="openshift-ingress/kube-auth-proxy-f974d944c-jkqks" Sep 11 14:20:28.617494 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.617469 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f974d944c-jkqks" Sep 11 14:20:28.901670 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:28.901645 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f974d944c-jkqks"] Sep 11 14:20:28.903740 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:20:28.903710 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71beef92_b67b_4ca2_82ec_803a0fb0f65c.slice/crio-9e1c10ca8fb00b1acb9104474b1bec550f07ee0a307be4b05a8daf3a883027db WatchSource:0}: Error finding container 9e1c10ca8fb00b1acb9104474b1bec550f07ee0a307be4b05a8daf3a883027db: Status 404 returned error can't find the container with id 9e1c10ca8fb00b1acb9104474b1bec550f07ee0a307be4b05a8daf3a883027db Sep 11 14:20:29.707204 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:29.707152 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f974d944c-jkqks" event={"ID":"71beef92-b67b-4ca2-82ec-803a0fb0f65c","Type":"ContainerStarted","Data":"9e1c10ca8fb00b1acb9104474b1bec550f07ee0a307be4b05a8daf3a883027db"} Sep 11 14:20:32.723118 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:32.723086 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f974d944c-jkqks" event={"ID":"71beef92-b67b-4ca2-82ec-803a0fb0f65c","Type":"ContainerStarted","Data":"41f4f5b9a9de525512bc46827d13407c349184f9c9008f8043b0da3bc3d84e99"} Sep 11 14:20:32.744773 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:32.744720 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-f974d944c-jkqks" podStartSLOduration=1.900439055 podStartE2EDuration="4.744706613s" totalImagesPullingTime="2.844267558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:32.743253535 +0000 UTC m=+345.080221729" watchObservedRunningTime="2026-09-11 14:20:32.744706613 +0000 UTC m=+345.081674805" Sep 11 14:20:33.280125 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:33.280087 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-g9xn5"] Sep 11 14:20:33.283519 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:33.283495 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" Sep 11 14:20:33.287122 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:33.287096 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qclpd\"" Sep 11 14:20:33.288179 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:33.288160 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 11 14:20:33.312154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:33.312128 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-g9xn5"] Sep 11 14:20:33.391861 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:33.391807 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff9827c8-e340-4b45-9b2f-45c3d53932ad-cert\") pod \"odh-model-controller-7998c67dc4-g9xn5\" (UID: \"ff9827c8-e340-4b45-9b2f-45c3d53932ad\") " pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" Sep 11 14:20:33.392009 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:33.391938 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6fqr\" (UniqueName: \"kubernetes.io/projected/ff9827c8-e340-4b45-9b2f-45c3d53932ad-kube-api-access-g6fqr\") pod \"odh-model-controller-7998c67dc4-g9xn5\" (UID: \"ff9827c8-e340-4b45-9b2f-45c3d53932ad\") " pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" Sep 11 14:20:33.493225 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:33.493191 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6fqr\" (UniqueName: \"kubernetes.io/projected/ff9827c8-e340-4b45-9b2f-45c3d53932ad-kube-api-access-g6fqr\") pod \"odh-model-controller-7998c67dc4-g9xn5\" (UID: \"ff9827c8-e340-4b45-9b2f-45c3d53932ad\") " pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" Sep 11 14:20:33.493374 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:33.493255 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff9827c8-e340-4b45-9b2f-45c3d53932ad-cert\") pod \"odh-model-controller-7998c67dc4-g9xn5\" (UID: \"ff9827c8-e340-4b45-9b2f-45c3d53932ad\") " pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" Sep 11 14:20:33.493374 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:20:33.493346 2573 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 11 14:20:33.493460 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:20:33.493405 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff9827c8-e340-4b45-9b2f-45c3d53932ad-cert podName:ff9827c8-e340-4b45-9b2f-45c3d53932ad nodeName:}" failed. No retries permitted until 2026-09-11 14:20:33.993388905 +0000 UTC m=+346.330357080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ff9827c8-e340-4b45-9b2f-45c3d53932ad-cert") pod "odh-model-controller-7998c67dc4-g9xn5" (UID: "ff9827c8-e340-4b45-9b2f-45c3d53932ad") : secret "odh-model-controller-webhook-cert" not found Sep 11 14:20:33.506501 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:33.506475 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6fqr\" (UniqueName: \"kubernetes.io/projected/ff9827c8-e340-4b45-9b2f-45c3d53932ad-kube-api-access-g6fqr\") pod \"odh-model-controller-7998c67dc4-g9xn5\" (UID: \"ff9827c8-e340-4b45-9b2f-45c3d53932ad\") " pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" Sep 11 14:20:33.997521 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:33.997484 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff9827c8-e340-4b45-9b2f-45c3d53932ad-cert\") pod \"odh-model-controller-7998c67dc4-g9xn5\" (UID: \"ff9827c8-e340-4b45-9b2f-45c3d53932ad\") " pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" Sep 11 14:20:33.999963 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:33.999937 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff9827c8-e340-4b45-9b2f-45c3d53932ad-cert\") pod \"odh-model-controller-7998c67dc4-g9xn5\" (UID: \"ff9827c8-e340-4b45-9b2f-45c3d53932ad\") " pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" Sep 11 14:20:34.133560 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:34.133525 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2lfxz"] Sep 11 14:20:34.137060 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:34.137044 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2lfxz" Sep 11 14:20:34.149448 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:34.149428 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 11 14:20:34.178919 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:34.178888 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2lfxz"] Sep 11 14:20:34.194426 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:34.194403 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" Sep 11 14:20:34.199321 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:34.199301 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shxz6\" (UniqueName: \"kubernetes.io/projected/1c819d4e-3764-4849-aa8d-b6f5b5385124-kube-api-access-shxz6\") pod \"model-serving-api-54d89bd5fc-2lfxz\" (UID: \"1c819d4e-3764-4849-aa8d-b6f5b5385124\") " pod="opendatahub/model-serving-api-54d89bd5fc-2lfxz" Sep 11 14:20:34.199426 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:34.199355 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c819d4e-3764-4849-aa8d-b6f5b5385124-tls-certs\") pod \"model-serving-api-54d89bd5fc-2lfxz\" (UID: \"1c819d4e-3764-4849-aa8d-b6f5b5385124\") " pod="opendatahub/model-serving-api-54d89bd5fc-2lfxz" Sep 11 14:20:34.300243 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:34.300178 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c819d4e-3764-4849-aa8d-b6f5b5385124-tls-certs\") pod \"model-serving-api-54d89bd5fc-2lfxz\" (UID: \"1c819d4e-3764-4849-aa8d-b6f5b5385124\") " pod="opendatahub/model-serving-api-54d89bd5fc-2lfxz" Sep 11 14:20:34.300356 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:34.300244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shxz6\" (UniqueName: \"kubernetes.io/projected/1c819d4e-3764-4849-aa8d-b6f5b5385124-kube-api-access-shxz6\") pod \"model-serving-api-54d89bd5fc-2lfxz\" (UID: \"1c819d4e-3764-4849-aa8d-b6f5b5385124\") " pod="opendatahub/model-serving-api-54d89bd5fc-2lfxz" Sep 11 14:20:34.302524 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:34.302494 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c819d4e-3764-4849-aa8d-b6f5b5385124-tls-certs\") pod \"model-serving-api-54d89bd5fc-2lfxz\" (UID: \"1c819d4e-3764-4849-aa8d-b6f5b5385124\") " pod="opendatahub/model-serving-api-54d89bd5fc-2lfxz" Sep 11 14:20:34.311722 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:34.311701 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shxz6\" (UniqueName: \"kubernetes.io/projected/1c819d4e-3764-4849-aa8d-b6f5b5385124-kube-api-access-shxz6\") pod \"model-serving-api-54d89bd5fc-2lfxz\" (UID: \"1c819d4e-3764-4849-aa8d-b6f5b5385124\") " pod="opendatahub/model-serving-api-54d89bd5fc-2lfxz" Sep 11 14:20:34.446852 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:34.446807 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2lfxz" Sep 11 14:20:34.472867 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:34.472821 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-g9xn5"] Sep 11 14:20:34.475069 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:20:34.475041 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9827c8_e340_4b45_9b2f_45c3d53932ad.slice/crio-55bc5a700ab37f8c9f48eac3be979a2476697e59dad06c2de4e4d38705adf777 WatchSource:0}: Error finding container 55bc5a700ab37f8c9f48eac3be979a2476697e59dad06c2de4e4d38705adf777: Status 404 returned error can't find the container with id 55bc5a700ab37f8c9f48eac3be979a2476697e59dad06c2de4e4d38705adf777 Sep 11 14:20:34.730166 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:34.730137 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2lfxz"] Sep 11 14:20:34.732077 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:20:34.732050 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c819d4e_3764_4849_aa8d_b6f5b5385124.slice/crio-bac88bbdaa3f287336670f59e7c72909b8ab97909f1ad28cc382468a30d0b522 WatchSource:0}: Error finding container bac88bbdaa3f287336670f59e7c72909b8ab97909f1ad28cc382468a30d0b522: Status 404 returned error can't find the container with id bac88bbdaa3f287336670f59e7c72909b8ab97909f1ad28cc382468a30d0b522 Sep 11 14:20:34.734357 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:34.734335 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" event={"ID":"ff9827c8-e340-4b45-9b2f-45c3d53932ad","Type":"ContainerStarted","Data":"55bc5a700ab37f8c9f48eac3be979a2476697e59dad06c2de4e4d38705adf777"} Sep 11 14:20:35.742702 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.742647 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2lfxz" event={"ID":"1c819d4e-3764-4849-aa8d-b6f5b5385124","Type":"ContainerStarted","Data":"bac88bbdaa3f287336670f59e7c72909b8ab97909f1ad28cc382468a30d0b522"} Sep 11 14:20:35.876156 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.876124 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6"] Sep 11 14:20:35.880332 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.880305 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:35.883134 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.883109 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 14:20:35.883553 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.883530 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-7km72\"" Sep 11 14:20:35.904058 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.904033 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6"] Sep 11 14:20:35.916002 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.915975 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:35.916125 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.916013 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11020d49-75f5-4c51-b532-7be4df336a1f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:35.916125 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.916065 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhs4s\" (UniqueName: \"kubernetes.io/projected/11020d49-75f5-4c51-b532-7be4df336a1f-kube-api-access-rhs4s\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:35.916125 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.916094 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11020d49-75f5-4c51-b532-7be4df336a1f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:35.916125 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.916122 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:35.916340 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.916187 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:35.916340 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.916287 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:35.916447 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.916338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11020d49-75f5-4c51-b532-7be4df336a1f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:35.916447 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.916389 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:35.978140 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.977656 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8"] Sep 11 14:20:35.981250 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.981167 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:35.999269 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:35.999211 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8"] Sep 11 14:20:36.018624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.017510 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011dd08d-5c4f-4801-b6ec-dff67cd76120-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.018624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.017564 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.018624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.017604 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011dd08d-5c4f-4801-b6ec-dff67cd76120-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.018624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.017644 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.018624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.017672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11020d49-75f5-4c51-b532-7be4df336a1f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.018624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.017698 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011dd08d-5c4f-4801-b6ec-dff67cd76120-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.018624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.017757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhs4s\" (UniqueName: \"kubernetes.io/projected/11020d49-75f5-4c51-b532-7be4df336a1f-kube-api-access-rhs4s\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.018624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.017782 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvhc6\" (UniqueName: \"kubernetes.io/projected/011dd08d-5c4f-4801-b6ec-dff67cd76120-kube-api-access-rvhc6\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.018624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.017812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11020d49-75f5-4c51-b532-7be4df336a1f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.018624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.017864 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011dd08d-5c4f-4801-b6ec-dff67cd76120-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.018624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.017891 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/011dd08d-5c4f-4801-b6ec-dff67cd76120-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.018624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.017946 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.018624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.018001 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.018624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.018037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011dd08d-5c4f-4801-b6ec-dff67cd76120-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.019468 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.018045 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.019468 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.018073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011dd08d-5c4f-4801-b6ec-dff67cd76120-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.019468 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.018113 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.019468 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.018146 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011dd08d-5c4f-4801-b6ec-dff67cd76120-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.019468 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.018188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11020d49-75f5-4c51-b532-7be4df336a1f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.019468 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.018318 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.019468 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.018414 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.019468 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.019057 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11020d49-75f5-4c51-b532-7be4df336a1f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.019468 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.019175 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.022632 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.022569 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11020d49-75f5-4c51-b532-7be4df336a1f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.023294 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.023270 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.028454 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.028400 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhs4s\" (UniqueName: \"kubernetes.io/projected/11020d49-75f5-4c51-b532-7be4df336a1f-kube-api-access-rhs4s\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.028967 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.028944 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11020d49-75f5-4c51-b532-7be4df336a1f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcd7db6\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.119077 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.119038 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011dd08d-5c4f-4801-b6ec-dff67cd76120-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.119235 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.119100 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011dd08d-5c4f-4801-b6ec-dff67cd76120-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.119235 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.119148 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvhc6\" (UniqueName: \"kubernetes.io/projected/011dd08d-5c4f-4801-b6ec-dff67cd76120-kube-api-access-rvhc6\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.119235 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.119177 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011dd08d-5c4f-4801-b6ec-dff67cd76120-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.119235 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.119201 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/011dd08d-5c4f-4801-b6ec-dff67cd76120-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.119478 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.119245 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011dd08d-5c4f-4801-b6ec-dff67cd76120-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.119478 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.119275 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011dd08d-5c4f-4801-b6ec-dff67cd76120-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.119478 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.119343 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011dd08d-5c4f-4801-b6ec-dff67cd76120-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.119478 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.119415 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011dd08d-5c4f-4801-b6ec-dff67cd76120-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.119795 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.119694 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011dd08d-5c4f-4801-b6ec-dff67cd76120-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.119880 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.119802 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011dd08d-5c4f-4801-b6ec-dff67cd76120-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.119880 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.119827 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011dd08d-5c4f-4801-b6ec-dff67cd76120-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.119880 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.119870 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011dd08d-5c4f-4801-b6ec-dff67cd76120-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.120032 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.119935 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011dd08d-5c4f-4801-b6ec-dff67cd76120-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.122694 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.122674 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011dd08d-5c4f-4801-b6ec-dff67cd76120-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.122813 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.122729 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011dd08d-5c4f-4801-b6ec-dff67cd76120-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.133037 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.133012 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvhc6\" (UniqueName: \"kubernetes.io/projected/011dd08d-5c4f-4801-b6ec-dff67cd76120-kube-api-access-rvhc6\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.134672 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.134650 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/011dd08d-5c4f-4801-b6ec-dff67cd76120-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8\" (UID: \"011dd08d-5c4f-4801-b6ec-dff67cd76120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:36.205176 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.205143 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:36.297979 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:36.297877 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:37.756130 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:37.756096 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" event={"ID":"ff9827c8-e340-4b45-9b2f-45c3d53932ad","Type":"ContainerStarted","Data":"9c66ebeb7a8897ee280a5e8748366ab9cedabbb4c75c9cca78cf40a11892d830"} Sep 11 14:20:37.756476 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:37.756239 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" Sep 11 14:20:37.758263 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:37.758238 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2lfxz" event={"ID":"1c819d4e-3764-4849-aa8d-b6f5b5385124","Type":"ContainerStarted","Data":"4226be15b467560b323da31acce051cc8def897b8323e40966697901d6465de9"} Sep 11 14:20:37.758380 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:37.758345 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-2lfxz" Sep 11 14:20:37.778294 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:37.778252 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" podStartSLOduration=1.603229422 podStartE2EDuration="4.77823845s" totalImagesPullingTime="3.175009028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:37.776102275 +0000 UTC m=+350.113070467" watchObservedRunningTime="2026-09-11 14:20:37.77823845 +0000 UTC m=+350.115206641" Sep 11 14:20:37.795319 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:37.795278 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-2lfxz" podStartSLOduration=0.930180037 podStartE2EDuration="3.795264778s" totalImagesPullingTime="2.865084741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:37.79448983 +0000 UTC m=+350.131458037" watchObservedRunningTime="2026-09-11 14:20:37.795264778 +0000 UTC m=+350.132232968" Sep 11 14:20:37.875949 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:37.875925 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8"] Sep 11 14:20:37.876786 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:20:37.876762 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011dd08d_5c4f_4801_b6ec_dff67cd76120.slice/crio-c6d48769ec87d8de97eb3427cee2a022f9b5187795f84d40c04dfc0a67d3b720 WatchSource:0}: Error finding container c6d48769ec87d8de97eb3427cee2a022f9b5187795f84d40c04dfc0a67d3b720: Status 404 returned error can't find the container with id c6d48769ec87d8de97eb3427cee2a022f9b5187795f84d40c04dfc0a67d3b720 Sep 11 14:20:37.887854 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:37.887816 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6"] Sep 11 14:20:37.889007 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:20:37.888988 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11020d49_75f5_4c51_b532_7be4df336a1f.slice/crio-3441bac51a4a80767323412369a12d41e33d80111dd4facfa5a64e778c598aa7 WatchSource:0}: Error finding container 3441bac51a4a80767323412369a12d41e33d80111dd4facfa5a64e778c598aa7: Status 404 returned error can't find the container with id 3441bac51a4a80767323412369a12d41e33d80111dd4facfa5a64e778c598aa7 Sep 11 14:20:38.763963 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:38.763925 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" event={"ID":"11020d49-75f5-4c51-b532-7be4df336a1f","Type":"ContainerStarted","Data":"3441bac51a4a80767323412369a12d41e33d80111dd4facfa5a64e778c598aa7"} Sep 11 14:20:38.765095 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:38.765074 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" event={"ID":"011dd08d-5c4f-4801-b6ec-dff67cd76120","Type":"ContainerStarted","Data":"c6d48769ec87d8de97eb3427cee2a022f9b5187795f84d40c04dfc0a67d3b720"} Sep 11 14:20:39.784368 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.784335 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf"] Sep 11 14:20:39.788323 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.788292 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:20:39.792217 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.792192 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cxdzr\"" Sep 11 14:20:39.793199 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.793131 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 11 14:20:39.794179 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.794154 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 11 14:20:39.795813 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.795618 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 11 14:20:39.795929 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.795892 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:20:39.805237 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.805186 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 11 14:20:39.833034 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.832997 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf"] Sep 11 14:20:39.853503 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.853471 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxm8f\" (UniqueName: \"kubernetes.io/projected/520952ba-90cd-4bb1-818f-a5aeb4c54de4-kube-api-access-bxm8f\") pod \"lws-controller-manager-7c4cd778db-mjktf\" (UID: \"520952ba-90cd-4bb1-818f-a5aeb4c54de4\") " pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:20:39.853636 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.853518 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/520952ba-90cd-4bb1-818f-a5aeb4c54de4-metrics-cert\") pod \"lws-controller-manager-7c4cd778db-mjktf\" (UID: \"520952ba-90cd-4bb1-818f-a5aeb4c54de4\") " pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:20:39.853636 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.853622 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/520952ba-90cd-4bb1-818f-a5aeb4c54de4-manager-config\") pod \"lws-controller-manager-7c4cd778db-mjktf\" (UID: \"520952ba-90cd-4bb1-818f-a5aeb4c54de4\") " pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:20:39.853822 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.853781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/520952ba-90cd-4bb1-818f-a5aeb4c54de4-cert\") pod \"lws-controller-manager-7c4cd778db-mjktf\" (UID: \"520952ba-90cd-4bb1-818f-a5aeb4c54de4\") " pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:20:39.955288 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.955247 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxm8f\" (UniqueName: \"kubernetes.io/projected/520952ba-90cd-4bb1-818f-a5aeb4c54de4-kube-api-access-bxm8f\") pod \"lws-controller-manager-7c4cd778db-mjktf\" (UID: \"520952ba-90cd-4bb1-818f-a5aeb4c54de4\") " pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:20:39.955450 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.955307 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/520952ba-90cd-4bb1-818f-a5aeb4c54de4-metrics-cert\") pod \"lws-controller-manager-7c4cd778db-mjktf\" (UID: \"520952ba-90cd-4bb1-818f-a5aeb4c54de4\") " pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:20:39.955450 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.955416 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/520952ba-90cd-4bb1-818f-a5aeb4c54de4-manager-config\") pod \"lws-controller-manager-7c4cd778db-mjktf\" (UID: \"520952ba-90cd-4bb1-818f-a5aeb4c54de4\") " pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:20:39.955583 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.955527 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/520952ba-90cd-4bb1-818f-a5aeb4c54de4-cert\") pod \"lws-controller-manager-7c4cd778db-mjktf\" (UID: \"520952ba-90cd-4bb1-818f-a5aeb4c54de4\") " pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:20:39.956378 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.956347 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/520952ba-90cd-4bb1-818f-a5aeb4c54de4-manager-config\") pod \"lws-controller-manager-7c4cd778db-mjktf\" (UID: \"520952ba-90cd-4bb1-818f-a5aeb4c54de4\") " pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:20:39.958606 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.958580 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/520952ba-90cd-4bb1-818f-a5aeb4c54de4-cert\") pod \"lws-controller-manager-7c4cd778db-mjktf\" (UID: \"520952ba-90cd-4bb1-818f-a5aeb4c54de4\") " pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:20:39.958722 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.958669 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/520952ba-90cd-4bb1-818f-a5aeb4c54de4-metrics-cert\") pod \"lws-controller-manager-7c4cd778db-mjktf\" (UID: \"520952ba-90cd-4bb1-818f-a5aeb4c54de4\") " pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:20:39.966053 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:39.966031 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxm8f\" (UniqueName: \"kubernetes.io/projected/520952ba-90cd-4bb1-818f-a5aeb4c54de4-kube-api-access-bxm8f\") pod \"lws-controller-manager-7c4cd778db-mjktf\" (UID: \"520952ba-90cd-4bb1-818f-a5aeb4c54de4\") " pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:20:40.101540 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:40.101303 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:20:40.417955 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:40.417927 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf"] Sep 11 14:20:41.250058 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:20:41.250013 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520952ba_90cd_4bb1_818f_a5aeb4c54de4.slice/crio-f7ac1c4c844beb98087465d52abcc22ab0d8701c90b5eedec8bc5cc5a5031b78 WatchSource:0}: Error finding container f7ac1c4c844beb98087465d52abcc22ab0d8701c90b5eedec8bc5cc5a5031b78: Status 404 returned error can't find the container with id f7ac1c4c844beb98087465d52abcc22ab0d8701c90b5eedec8bc5cc5a5031b78 Sep 11 14:20:41.326401 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:41.326364 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:20:41.326493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:41.326437 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:20:41.326493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:41.326465 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:20:41.332550 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:41.332524 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:20:41.332597 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:41.332581 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:20:41.332632 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:41.332608 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:20:41.788161 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:41.788108 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" event={"ID":"11020d49-75f5-4c51-b532-7be4df336a1f","Type":"ContainerStarted","Data":"68b851a14ca0d5f8be2a9759d3e13ec1bdc1d430648b978df19bd81c81d1ee74"} Sep 11 14:20:41.789752 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:41.789722 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" event={"ID":"520952ba-90cd-4bb1-818f-a5aeb4c54de4","Type":"ContainerStarted","Data":"f7ac1c4c844beb98087465d52abcc22ab0d8701c90b5eedec8bc5cc5a5031b78"} Sep 11 14:20:41.791572 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:41.791540 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" event={"ID":"011dd08d-5c4f-4801-b6ec-dff67cd76120","Type":"ContainerStarted","Data":"9cea04de80575b4ed4f2deffeaa3a995b5960a06bb1e84bbeec593aa690dc9b9"} Sep 11 14:20:41.886312 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:41.886261 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" podStartSLOduration=3.451040942 podStartE2EDuration="6.886242612s" totalImagesPullingTime="3.43520167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:41.875880569 +0000 UTC m=+354.212848760" watchObservedRunningTime="2026-09-11 14:20:41.886242612 +0000 UTC m=+354.223210806" Sep 11 14:20:41.975025 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:41.974967 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" podStartSLOduration=3.521681126 podStartE2EDuration="6.974926967s" totalImagesPullingTime="3.453245841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:41.974877291 +0000 UTC m=+354.311845486" watchObservedRunningTime="2026-09-11 14:20:41.974926967 +0000 UTC m=+354.311895158" Sep 11 14:20:42.205568 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:42.205528 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:42.207058 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:42.207021 2573 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 11 14:20:42.207194 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:42.207082 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" podUID="11020d49-75f5-4c51-b532-7be4df336a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 11 14:20:42.298664 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:42.298625 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:42.304763 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:42.304739 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:42.798218 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:42.798173 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" event={"ID":"520952ba-90cd-4bb1-818f-a5aeb4c54de4","Type":"ContainerStarted","Data":"f69b867e5d58ac2305258666897acfaed25a7c524ed559b0154c74e826081464"} Sep 11 14:20:42.798424 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:42.798409 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:42.799545 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:42.799523 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8" Sep 11 14:20:42.973618 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:42.973547 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" podStartSLOduration=2.6599567779999997 podStartE2EDuration="3.973523751s" totalImagesPullingTime="1.313566973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:42.863121772 +0000 UTC m=+355.200089964" watchObservedRunningTime="2026-09-11 14:20:42.973523751 +0000 UTC m=+355.310491945" Sep 11 14:20:43.206463 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:43.206428 2573 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 11 14:20:43.206620 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:43.206490 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" podUID="11020d49-75f5-4c51-b532-7be4df336a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 11 14:20:43.285987 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:43.285911 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6"] Sep 11 14:20:43.803450 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:43.803414 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:20:44.206162 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:44.206108 2573 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 11 14:20:44.206162 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:44.206159 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" podUID="11020d49-75f5-4c51-b532-7be4df336a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 11 14:20:44.807955 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:44.807913 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" podUID="11020d49-75f5-4c51-b532-7be4df336a1f" containerName="istio-proxy" containerID="cri-o://68b851a14ca0d5f8be2a9759d3e13ec1bdc1d430648b978df19bd81c81d1ee74" gracePeriod=30 Sep 11 14:20:48.767782 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:48.767746 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" Sep 11 14:20:48.770442 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:48.770422 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-2lfxz" Sep 11 14:20:50.061569 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.061541 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:50.156079 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.156051 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-istio-envoy\") pod \"11020d49-75f5-4c51-b532-7be4df336a1f\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " Sep 11 14:20:50.156079 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.156081 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-workload-certs\") pod \"11020d49-75f5-4c51-b532-7be4df336a1f\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " Sep 11 14:20:50.156285 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.156115 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11020d49-75f5-4c51-b532-7be4df336a1f-istiod-ca-cert\") pod \"11020d49-75f5-4c51-b532-7be4df336a1f\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " Sep 11 14:20:50.156285 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.156148 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-credential-socket\") pod \"11020d49-75f5-4c51-b532-7be4df336a1f\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " Sep 11 14:20:50.156285 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.156165 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-istio-data\") pod \"11020d49-75f5-4c51-b532-7be4df336a1f\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " Sep 11 14:20:50.156285 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.156188 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11020d49-75f5-4c51-b532-7be4df336a1f-istio-podinfo\") pod \"11020d49-75f5-4c51-b532-7be4df336a1f\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " Sep 11 14:20:50.156285 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.156214 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-workload-socket\") pod \"11020d49-75f5-4c51-b532-7be4df336a1f\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " Sep 11 14:20:50.156285 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.156263 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhs4s\" (UniqueName: \"kubernetes.io/projected/11020d49-75f5-4c51-b532-7be4df336a1f-kube-api-access-rhs4s\") pod \"11020d49-75f5-4c51-b532-7be4df336a1f\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " Sep 11 14:20:50.156285 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.156287 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11020d49-75f5-4c51-b532-7be4df336a1f-istio-token\") pod \"11020d49-75f5-4c51-b532-7be4df336a1f\" (UID: \"11020d49-75f5-4c51-b532-7be4df336a1f\") " Sep 11 14:20:50.157140 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.156372 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11020d49-75f5-4c51-b532-7be4df336a1f" (UID: "11020d49-75f5-4c51-b532-7be4df336a1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:20:50.157140 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.156500 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-istio-data" (OuterVolumeSpecName: "istio-data") pod "11020d49-75f5-4c51-b532-7be4df336a1f" (UID: "11020d49-75f5-4c51-b532-7be4df336a1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:20:50.157140 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.156614 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11020d49-75f5-4c51-b532-7be4df336a1f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11020d49-75f5-4c51-b532-7be4df336a1f" (UID: "11020d49-75f5-4c51-b532-7be4df336a1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:20:50.157140 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.156671 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:20:50.157140 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.156691 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:20:50.157140 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.156723 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11020d49-75f5-4c51-b532-7be4df336a1f" (UID: "11020d49-75f5-4c51-b532-7be4df336a1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:20:50.157140 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.157086 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11020d49-75f5-4c51-b532-7be4df336a1f" (UID: "11020d49-75f5-4c51-b532-7be4df336a1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:20:50.158800 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.158773 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11020d49-75f5-4c51-b532-7be4df336a1f" (UID: "11020d49-75f5-4c51-b532-7be4df336a1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:20:50.158941 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.158798 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11020d49-75f5-4c51-b532-7be4df336a1f-istio-token" (OuterVolumeSpecName: "istio-token") pod "11020d49-75f5-4c51-b532-7be4df336a1f" (UID: "11020d49-75f5-4c51-b532-7be4df336a1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:20:50.158941 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.158885 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11020d49-75f5-4c51-b532-7be4df336a1f-kube-api-access-rhs4s" (OuterVolumeSpecName: "kube-api-access-rhs4s") pod "11020d49-75f5-4c51-b532-7be4df336a1f" (UID: "11020d49-75f5-4c51-b532-7be4df336a1f"). InnerVolumeSpecName "kube-api-access-rhs4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:20:50.159027 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.159017 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11020d49-75f5-4c51-b532-7be4df336a1f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11020d49-75f5-4c51-b532-7be4df336a1f" (UID: "11020d49-75f5-4c51-b532-7be4df336a1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:20:50.257655 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.257631 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:20:50.257655 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.257653 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11020d49-75f5-4c51-b532-7be4df336a1f-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:20:50.257873 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.257663 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:20:50.257873 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.257673 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11020d49-75f5-4c51-b532-7be4df336a1f-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:20:50.257873 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.257682 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11020d49-75f5-4c51-b532-7be4df336a1f-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:20:50.257873 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.257691 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhs4s\" (UniqueName: \"kubernetes.io/projected/11020d49-75f5-4c51-b532-7be4df336a1f-kube-api-access-rhs4s\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:20:50.257873 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.257699 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11020d49-75f5-4c51-b532-7be4df336a1f-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:20:50.842278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.842240 2573 generic.go:356] "Generic (PLEG): container finished" podID="11020d49-75f5-4c51-b532-7be4df336a1f" containerID="68b851a14ca0d5f8be2a9759d3e13ec1bdc1d430648b978df19bd81c81d1ee74" exitCode=0 Sep 11 14:20:50.842514 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.842288 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" Sep 11 14:20:50.842514 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.842310 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" event={"ID":"11020d49-75f5-4c51-b532-7be4df336a1f","Type":"ContainerDied","Data":"68b851a14ca0d5f8be2a9759d3e13ec1bdc1d430648b978df19bd81c81d1ee74"} Sep 11 14:20:50.842514 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.842361 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6" event={"ID":"11020d49-75f5-4c51-b532-7be4df336a1f","Type":"ContainerDied","Data":"3441bac51a4a80767323412369a12d41e33d80111dd4facfa5a64e778c598aa7"} Sep 11 14:20:50.842514 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.842391 2573 scope.go:117] "RemoveContainer" containerID="68b851a14ca0d5f8be2a9759d3e13ec1bdc1d430648b978df19bd81c81d1ee74" Sep 11 14:20:50.850902 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.850880 2573 scope.go:117] "RemoveContainer" containerID="68b851a14ca0d5f8be2a9759d3e13ec1bdc1d430648b978df19bd81c81d1ee74" Sep 11 14:20:50.851187 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:20:50.851166 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b851a14ca0d5f8be2a9759d3e13ec1bdc1d430648b978df19bd81c81d1ee74\": container with ID starting with 68b851a14ca0d5f8be2a9759d3e13ec1bdc1d430648b978df19bd81c81d1ee74 not found: ID does not exist" containerID="68b851a14ca0d5f8be2a9759d3e13ec1bdc1d430648b978df19bd81c81d1ee74" Sep 11 14:20:50.851240 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.851195 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b851a14ca0d5f8be2a9759d3e13ec1bdc1d430648b978df19bd81c81d1ee74"} err="failed to get container status \"68b851a14ca0d5f8be2a9759d3e13ec1bdc1d430648b978df19bd81c81d1ee74\": rpc error: code = NotFound desc = could not find container \"68b851a14ca0d5f8be2a9759d3e13ec1bdc1d430648b978df19bd81c81d1ee74\": container with ID starting with 68b851a14ca0d5f8be2a9759d3e13ec1bdc1d430648b978df19bd81c81d1ee74 not found: ID does not exist" Sep 11 14:20:50.880930 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.880903 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6"] Sep 11 14:20:50.904707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:50.904675 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcd7db6"] Sep 11 14:20:52.187524 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:52.187489 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11020d49-75f5-4c51-b532-7be4df336a1f" path="/var/lib/kubelet/pods/11020d49-75f5-4c51-b532-7be4df336a1f/volumes" Sep 11 14:20:54.810018 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:20:54.809988 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:21:40.168772 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.168740 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5"] Sep 11 14:21:40.169193 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.169106 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11020d49-75f5-4c51-b532-7be4df336a1f" containerName="istio-proxy" Sep 11 14:21:40.169193 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.169117 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="11020d49-75f5-4c51-b532-7be4df336a1f" containerName="istio-proxy" Sep 11 14:21:40.169193 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.169179 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="11020d49-75f5-4c51-b532-7be4df336a1f" containerName="istio-proxy" Sep 11 14:21:40.171792 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.171774 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" Sep 11 14:21:40.174664 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.174643 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:21:40.175645 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.175621 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:21:40.175761 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.175648 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4brs6\"" Sep 11 14:21:40.188879 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.188858 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5"] Sep 11 14:21:40.268792 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.268757 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht"] Sep 11 14:21:40.271438 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.271422 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" Sep 11 14:21:40.282923 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.282902 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht"] Sep 11 14:21:40.286077 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.286055 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a538801-cd1b-48a9-8855-baeefff75ec7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5\" (UID: \"3a538801-cd1b-48a9-8855-baeefff75ec7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" Sep 11 14:21:40.286188 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.286084 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs4bb\" (UniqueName: \"kubernetes.io/projected/3a538801-cd1b-48a9-8855-baeefff75ec7-kube-api-access-cs4bb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5\" (UID: \"3a538801-cd1b-48a9-8855-baeefff75ec7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" Sep 11 14:21:40.286188 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.286102 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a538801-cd1b-48a9-8855-baeefff75ec7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5\" (UID: \"3a538801-cd1b-48a9-8855-baeefff75ec7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" Sep 11 14:21:40.355380 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.355351 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5"] Sep 11 14:21:40.358030 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.358015 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" Sep 11 14:21:40.373182 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.373163 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5"] Sep 11 14:21:40.387252 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.387231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a538801-cd1b-48a9-8855-baeefff75ec7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5\" (UID: \"3a538801-cd1b-48a9-8855-baeefff75ec7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" Sep 11 14:21:40.387367 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.387260 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs4bb\" (UniqueName: \"kubernetes.io/projected/3a538801-cd1b-48a9-8855-baeefff75ec7-kube-api-access-cs4bb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5\" (UID: \"3a538801-cd1b-48a9-8855-baeefff75ec7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" Sep 11 14:21:40.387367 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.387277 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a538801-cd1b-48a9-8855-baeefff75ec7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5\" (UID: \"3a538801-cd1b-48a9-8855-baeefff75ec7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" Sep 11 14:21:40.387367 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.387295 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f401b3e-099a-414d-bd11-a8ae40d61464-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht\" (UID: \"8f401b3e-099a-414d-bd11-a8ae40d61464\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" Sep 11 14:21:40.387367 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.387341 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zttks\" (UniqueName: \"kubernetes.io/projected/8f401b3e-099a-414d-bd11-a8ae40d61464-kube-api-access-zttks\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht\" (UID: \"8f401b3e-099a-414d-bd11-a8ae40d61464\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" Sep 11 14:21:40.387601 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.387371 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f401b3e-099a-414d-bd11-a8ae40d61464-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht\" (UID: \"8f401b3e-099a-414d-bd11-a8ae40d61464\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" Sep 11 14:21:40.387601 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.387580 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a538801-cd1b-48a9-8855-baeefff75ec7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5\" (UID: \"3a538801-cd1b-48a9-8855-baeefff75ec7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" Sep 11 14:21:40.387701 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.387647 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a538801-cd1b-48a9-8855-baeefff75ec7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5\" (UID: \"3a538801-cd1b-48a9-8855-baeefff75ec7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" Sep 11 14:21:40.401227 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.401199 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs4bb\" (UniqueName: \"kubernetes.io/projected/3a538801-cd1b-48a9-8855-baeefff75ec7-kube-api-access-cs4bb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5\" (UID: \"3a538801-cd1b-48a9-8855-baeefff75ec7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" Sep 11 14:21:40.481190 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.481129 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" Sep 11 14:21:40.488324 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.488302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5\" (UID: \"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" Sep 11 14:21:40.488418 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.488338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5\" (UID: \"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" Sep 11 14:21:40.488418 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.488360 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f401b3e-099a-414d-bd11-a8ae40d61464-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht\" (UID: \"8f401b3e-099a-414d-bd11-a8ae40d61464\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" Sep 11 14:21:40.488505 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.488461 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rh74\" (UniqueName: \"kubernetes.io/projected/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-kube-api-access-7rh74\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5\" (UID: \"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" Sep 11 14:21:40.488505 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.488489 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zttks\" (UniqueName: \"kubernetes.io/projected/8f401b3e-099a-414d-bd11-a8ae40d61464-kube-api-access-zttks\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht\" (UID: \"8f401b3e-099a-414d-bd11-a8ae40d61464\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" Sep 11 14:21:40.488582 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.488528 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f401b3e-099a-414d-bd11-a8ae40d61464-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht\" (UID: \"8f401b3e-099a-414d-bd11-a8ae40d61464\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" Sep 11 14:21:40.488660 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.488640 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f401b3e-099a-414d-bd11-a8ae40d61464-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht\" (UID: \"8f401b3e-099a-414d-bd11-a8ae40d61464\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" Sep 11 14:21:40.488885 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.488865 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f401b3e-099a-414d-bd11-a8ae40d61464-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht\" (UID: \"8f401b3e-099a-414d-bd11-a8ae40d61464\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" Sep 11 14:21:40.492945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.492923 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc"] Sep 11 14:21:40.496082 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.496056 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" Sep 11 14:21:40.505950 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.505902 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zttks\" (UniqueName: \"kubernetes.io/projected/8f401b3e-099a-414d-bd11-a8ae40d61464-kube-api-access-zttks\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht\" (UID: \"8f401b3e-099a-414d-bd11-a8ae40d61464\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" Sep 11 14:21:40.507819 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.507797 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc"] Sep 11 14:21:40.580427 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.580398 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" Sep 11 14:21:40.589690 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.589666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5\" (UID: \"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" Sep 11 14:21:40.589801 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.589700 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/520ec79b-a25c-454c-a1e1-ba9572490cfd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc\" (UID: \"520ec79b-a25c-454c-a1e1-ba9572490cfd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" Sep 11 14:21:40.589801 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.589722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5\" (UID: \"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" Sep 11 14:21:40.589801 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.589754 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/520ec79b-a25c-454c-a1e1-ba9572490cfd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc\" (UID: \"520ec79b-a25c-454c-a1e1-ba9572490cfd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" Sep 11 14:21:40.590006 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.589800 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rh74\" (UniqueName: \"kubernetes.io/projected/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-kube-api-access-7rh74\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5\" (UID: \"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" Sep 11 14:21:40.590006 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.589821 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n9nj\" (UniqueName: \"kubernetes.io/projected/520ec79b-a25c-454c-a1e1-ba9572490cfd-kube-api-access-5n9nj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc\" (UID: \"520ec79b-a25c-454c-a1e1-ba9572490cfd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" Sep 11 14:21:40.590254 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.590091 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5\" (UID: \"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" Sep 11 14:21:40.590254 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.590105 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5\" (UID: \"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" Sep 11 14:21:40.599004 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.598981 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rh74\" (UniqueName: \"kubernetes.io/projected/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-kube-api-access-7rh74\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5\" (UID: \"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" Sep 11 14:21:40.667249 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.667220 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" Sep 11 14:21:40.690910 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.690885 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/520ec79b-a25c-454c-a1e1-ba9572490cfd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc\" (UID: \"520ec79b-a25c-454c-a1e1-ba9572490cfd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" Sep 11 14:21:40.691030 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.690922 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/520ec79b-a25c-454c-a1e1-ba9572490cfd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc\" (UID: \"520ec79b-a25c-454c-a1e1-ba9572490cfd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" Sep 11 14:21:40.691030 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.690987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n9nj\" (UniqueName: \"kubernetes.io/projected/520ec79b-a25c-454c-a1e1-ba9572490cfd-kube-api-access-5n9nj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc\" (UID: \"520ec79b-a25c-454c-a1e1-ba9572490cfd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" Sep 11 14:21:40.691293 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.691271 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/520ec79b-a25c-454c-a1e1-ba9572490cfd-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc\" (UID: \"520ec79b-a25c-454c-a1e1-ba9572490cfd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" Sep 11 14:21:40.691377 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.691327 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/520ec79b-a25c-454c-a1e1-ba9572490cfd-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc\" (UID: \"520ec79b-a25c-454c-a1e1-ba9572490cfd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" Sep 11 14:21:40.700132 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.700104 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n9nj\" (UniqueName: \"kubernetes.io/projected/520ec79b-a25c-454c-a1e1-ba9572490cfd-kube-api-access-5n9nj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc\" (UID: \"520ec79b-a25c-454c-a1e1-ba9572490cfd\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" Sep 11 14:21:40.756029 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.756004 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5"] Sep 11 14:21:40.757641 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:21:40.757613 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a538801_cd1b_48a9_8855_baeefff75ec7.slice/crio-b93eb76b5c2291f9dcdf5d23e121724ff8f6196b231f7121a08de279b1c5485c WatchSource:0}: Error finding container b93eb76b5c2291f9dcdf5d23e121724ff8f6196b231f7121a08de279b1c5485c: Status 404 returned error can't find the container with id b93eb76b5c2291f9dcdf5d23e121724ff8f6196b231f7121a08de279b1c5485c Sep 11 14:21:40.821343 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.821318 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" Sep 11 14:21:40.856395 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.856368 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht"] Sep 11 14:21:40.883049 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:21:40.883016 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f401b3e_099a_414d_bd11_a8ae40d61464.slice/crio-625465d3fae76a8c0cc143199176399b682f821d76bb11f479f9646df8bcf5aa WatchSource:0}: Error finding container 625465d3fae76a8c0cc143199176399b682f821d76bb11f479f9646df8bcf5aa: Status 404 returned error can't find the container with id 625465d3fae76a8c0cc143199176399b682f821d76bb11f479f9646df8bcf5aa Sep 11 14:21:40.939561 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:40.939538 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5"] Sep 11 14:21:40.940927 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:21:40.940904 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad8b04d_bc8d_4fc7_ab5e_49853abdcf12.slice/crio-707bfd61ec35f5ad3d04ee91b17a64d579a491aee95ca7142a89fe7ec178a5c5 WatchSource:0}: Error finding container 707bfd61ec35f5ad3d04ee91b17a64d579a491aee95ca7142a89fe7ec178a5c5: Status 404 returned error can't find the container with id 707bfd61ec35f5ad3d04ee91b17a64d579a491aee95ca7142a89fe7ec178a5c5 Sep 11 14:21:41.087558 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:41.087528 2573 generic.go:356] "Generic (PLEG): container finished" podID="9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12" containerID="7b3b19ad31309e12b92bc1cd9139b0ef2818e35b9e22dfc5b7ad6c40ff5e6cb7" exitCode=0 Sep 11 14:21:41.087675 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:41.087607 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" event={"ID":"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12","Type":"ContainerDied","Data":"7b3b19ad31309e12b92bc1cd9139b0ef2818e35b9e22dfc5b7ad6c40ff5e6cb7"} Sep 11 14:21:41.087675 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:41.087661 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" event={"ID":"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12","Type":"ContainerStarted","Data":"707bfd61ec35f5ad3d04ee91b17a64d579a491aee95ca7142a89fe7ec178a5c5"} Sep 11 14:21:41.089127 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:41.089107 2573 generic.go:356] "Generic (PLEG): container finished" podID="8f401b3e-099a-414d-bd11-a8ae40d61464" containerID="0430a704c9f64cd476089b5437afbc4440f6d760bb0e91e427f56447cb18afa6" exitCode=0 Sep 11 14:21:41.089211 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:41.089192 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" event={"ID":"8f401b3e-099a-414d-bd11-a8ae40d61464","Type":"ContainerDied","Data":"0430a704c9f64cd476089b5437afbc4440f6d760bb0e91e427f56447cb18afa6"} Sep 11 14:21:41.089281 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:41.089270 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" event={"ID":"8f401b3e-099a-414d-bd11-a8ae40d61464","Type":"ContainerStarted","Data":"625465d3fae76a8c0cc143199176399b682f821d76bb11f479f9646df8bcf5aa"} Sep 11 14:21:41.090878 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:41.090857 2573 generic.go:356] "Generic (PLEG): container finished" podID="3a538801-cd1b-48a9-8855-baeefff75ec7" containerID="b6d6e7fbe56a5eda12068c7bcd975209ffc13c91438439d4a5615ba0f9792476" exitCode=0 Sep 11 14:21:41.090970 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:41.090950 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" event={"ID":"3a538801-cd1b-48a9-8855-baeefff75ec7","Type":"ContainerDied","Data":"b6d6e7fbe56a5eda12068c7bcd975209ffc13c91438439d4a5615ba0f9792476"} Sep 11 14:21:41.091012 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:41.090990 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" event={"ID":"3a538801-cd1b-48a9-8855-baeefff75ec7","Type":"ContainerStarted","Data":"b93eb76b5c2291f9dcdf5d23e121724ff8f6196b231f7121a08de279b1c5485c"} Sep 11 14:21:41.098521 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:41.098494 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc"] Sep 11 14:21:41.099691 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:21:41.099663 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520ec79b_a25c_454c_a1e1_ba9572490cfd.slice/crio-ad985ed0c67f311f3c11c80b37723f1d45db64e079ef0cbeb6b566a752b03d12 WatchSource:0}: Error finding container ad985ed0c67f311f3c11c80b37723f1d45db64e079ef0cbeb6b566a752b03d12: Status 404 returned error can't find the container with id ad985ed0c67f311f3c11c80b37723f1d45db64e079ef0cbeb6b566a752b03d12 Sep 11 14:21:42.097460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:42.097429 2573 generic.go:356] "Generic (PLEG): container finished" podID="520ec79b-a25c-454c-a1e1-ba9572490cfd" containerID="01727df988cc215b97246df290b90696413555f180269c728e9f08e3b48359e0" exitCode=0 Sep 11 14:21:42.097858 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:42.097511 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" event={"ID":"520ec79b-a25c-454c-a1e1-ba9572490cfd","Type":"ContainerDied","Data":"01727df988cc215b97246df290b90696413555f180269c728e9f08e3b48359e0"} Sep 11 14:21:42.097858 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:42.097568 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" event={"ID":"520ec79b-a25c-454c-a1e1-ba9572490cfd","Type":"ContainerStarted","Data":"ad985ed0c67f311f3c11c80b37723f1d45db64e079ef0cbeb6b566a752b03d12"} Sep 11 14:21:42.099710 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:42.099684 2573 generic.go:356] "Generic (PLEG): container finished" podID="9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12" containerID="4607a660bd4d443acf8e4800813064e56847ce4289cd66deaad7446f31494e58" exitCode=0 Sep 11 14:21:42.099812 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:42.099770 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" event={"ID":"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12","Type":"ContainerDied","Data":"4607a660bd4d443acf8e4800813064e56847ce4289cd66deaad7446f31494e58"} Sep 11 14:21:42.101709 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:42.101687 2573 generic.go:356] "Generic (PLEG): container finished" podID="3a538801-cd1b-48a9-8855-baeefff75ec7" containerID="1a97338c23a64d7078e2d9d7ae650fb15363eadd6a4320a26e48a235434a0f1b" exitCode=0 Sep 11 14:21:42.101801 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:42.101735 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" event={"ID":"3a538801-cd1b-48a9-8855-baeefff75ec7","Type":"ContainerDied","Data":"1a97338c23a64d7078e2d9d7ae650fb15363eadd6a4320a26e48a235434a0f1b"} Sep 11 14:21:43.109466 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:43.109438 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" event={"ID":"520ec79b-a25c-454c-a1e1-ba9572490cfd","Type":"ContainerStarted","Data":"e77b00e728cc132989ad5bd9645664fa828f0b2e4c3b3cfaba5437cb6f22cb0f"} Sep 11 14:21:43.111639 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:43.111609 2573 generic.go:356] "Generic (PLEG): container finished" podID="9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12" containerID="d54c1abd440b32ec8bfdd7e3935118040ced603348c6cb953a3f18d1d2e4a3b0" exitCode=0 Sep 11 14:21:43.111729 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:43.111645 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" event={"ID":"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12","Type":"ContainerDied","Data":"d54c1abd440b32ec8bfdd7e3935118040ced603348c6cb953a3f18d1d2e4a3b0"} Sep 11 14:21:43.113630 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:43.113608 2573 generic.go:356] "Generic (PLEG): container finished" podID="8f401b3e-099a-414d-bd11-a8ae40d61464" containerID="fbd4638635bb06eff2951045c651898cde9f30102847012a02148832d7c55b2b" exitCode=0 Sep 11 14:21:43.113731 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:43.113685 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" event={"ID":"8f401b3e-099a-414d-bd11-a8ae40d61464","Type":"ContainerDied","Data":"fbd4638635bb06eff2951045c651898cde9f30102847012a02148832d7c55b2b"} Sep 11 14:21:43.116142 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:43.116122 2573 generic.go:356] "Generic (PLEG): container finished" podID="3a538801-cd1b-48a9-8855-baeefff75ec7" containerID="46a926da00f15983d740f604577d3326468c35dc14b3ef9202e1ce83f131bdec" exitCode=0 Sep 11 14:21:43.116231 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:43.116190 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" event={"ID":"3a538801-cd1b-48a9-8855-baeefff75ec7","Type":"ContainerDied","Data":"46a926da00f15983d740f604577d3326468c35dc14b3ef9202e1ce83f131bdec"} Sep 11 14:21:44.123495 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.123462 2573 generic.go:356] "Generic (PLEG): container finished" podID="520ec79b-a25c-454c-a1e1-ba9572490cfd" containerID="e77b00e728cc132989ad5bd9645664fa828f0b2e4c3b3cfaba5437cb6f22cb0f" exitCode=0 Sep 11 14:21:44.123971 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.123555 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" event={"ID":"520ec79b-a25c-454c-a1e1-ba9572490cfd","Type":"ContainerDied","Data":"e77b00e728cc132989ad5bd9645664fa828f0b2e4c3b3cfaba5437cb6f22cb0f"} Sep 11 14:21:44.125906 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.125885 2573 generic.go:356] "Generic (PLEG): container finished" podID="8f401b3e-099a-414d-bd11-a8ae40d61464" containerID="e159ff0c4e7abda2021920e9db4fd0d43c01657328ba62aa16df03920b6db155" exitCode=0 Sep 11 14:21:44.125989 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.125975 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" event={"ID":"8f401b3e-099a-414d-bd11-a8ae40d61464","Type":"ContainerDied","Data":"e159ff0c4e7abda2021920e9db4fd0d43c01657328ba62aa16df03920b6db155"} Sep 11 14:21:44.326554 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.326534 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" Sep 11 14:21:44.338301 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.338277 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" Sep 11 14:21:44.422105 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.422032 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a538801-cd1b-48a9-8855-baeefff75ec7-util\") pod \"3a538801-cd1b-48a9-8855-baeefff75ec7\" (UID: \"3a538801-cd1b-48a9-8855-baeefff75ec7\") " Sep 11 14:21:44.422105 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.422084 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a538801-cd1b-48a9-8855-baeefff75ec7-bundle\") pod \"3a538801-cd1b-48a9-8855-baeefff75ec7\" (UID: \"3a538801-cd1b-48a9-8855-baeefff75ec7\") " Sep 11 14:21:44.422295 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.422126 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-bundle\") pod \"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12\" (UID: \"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12\") " Sep 11 14:21:44.422295 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.422186 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-util\") pod \"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12\" (UID: \"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12\") " Sep 11 14:21:44.422295 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.422225 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rh74\" (UniqueName: \"kubernetes.io/projected/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-kube-api-access-7rh74\") pod \"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12\" (UID: \"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12\") " Sep 11 14:21:44.422295 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.422280 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cs4bb\" (UniqueName: \"kubernetes.io/projected/3a538801-cd1b-48a9-8855-baeefff75ec7-kube-api-access-cs4bb\") pod \"3a538801-cd1b-48a9-8855-baeefff75ec7\" (UID: \"3a538801-cd1b-48a9-8855-baeefff75ec7\") " Sep 11 14:21:44.422910 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.422874 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a538801-cd1b-48a9-8855-baeefff75ec7-bundle" (OuterVolumeSpecName: "bundle") pod "3a538801-cd1b-48a9-8855-baeefff75ec7" (UID: "3a538801-cd1b-48a9-8855-baeefff75ec7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:21:44.423119 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.423085 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-bundle" (OuterVolumeSpecName: "bundle") pod "9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12" (UID: "9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:21:44.424451 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.424421 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a538801-cd1b-48a9-8855-baeefff75ec7-kube-api-access-cs4bb" (OuterVolumeSpecName: "kube-api-access-cs4bb") pod "3a538801-cd1b-48a9-8855-baeefff75ec7" (UID: "3a538801-cd1b-48a9-8855-baeefff75ec7"). InnerVolumeSpecName "kube-api-access-cs4bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:21:44.424653 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.424635 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-kube-api-access-7rh74" (OuterVolumeSpecName: "kube-api-access-7rh74") pod "9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12" (UID: "9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12"). InnerVolumeSpecName "kube-api-access-7rh74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:21:44.428033 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.428013 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-util" (OuterVolumeSpecName: "util") pod "9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12" (UID: "9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:21:44.429026 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.429006 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a538801-cd1b-48a9-8855-baeefff75ec7-util" (OuterVolumeSpecName: "util") pod "3a538801-cd1b-48a9-8855-baeefff75ec7" (UID: "3a538801-cd1b-48a9-8855-baeefff75ec7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:21:44.523207 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.523179 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs4bb\" (UniqueName: \"kubernetes.io/projected/3a538801-cd1b-48a9-8855-baeefff75ec7-kube-api-access-cs4bb\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:21:44.523207 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.523203 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a538801-cd1b-48a9-8855-baeefff75ec7-util\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:21:44.523207 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.523213 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a538801-cd1b-48a9-8855-baeefff75ec7-bundle\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:21:44.523403 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.523221 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-bundle\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:21:44.523403 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.523229 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-util\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:21:44.523403 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:44.523238 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rh74\" (UniqueName: \"kubernetes.io/projected/9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12-kube-api-access-7rh74\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:21:45.134148 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.134111 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" Sep 11 14:21:45.134148 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.134126 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6lfb5" event={"ID":"9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12","Type":"ContainerDied","Data":"707bfd61ec35f5ad3d04ee91b17a64d579a491aee95ca7142a89fe7ec178a5c5"} Sep 11 14:21:45.134610 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.134211 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="707bfd61ec35f5ad3d04ee91b17a64d579a491aee95ca7142a89fe7ec178a5c5" Sep 11 14:21:45.136058 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.136041 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" Sep 11 14:21:45.136180 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.136076 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cznbr5" event={"ID":"3a538801-cd1b-48a9-8855-baeefff75ec7","Type":"ContainerDied","Data":"b93eb76b5c2291f9dcdf5d23e121724ff8f6196b231f7121a08de279b1c5485c"} Sep 11 14:21:45.136180 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.136122 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b93eb76b5c2291f9dcdf5d23e121724ff8f6196b231f7121a08de279b1c5485c" Sep 11 14:21:45.138149 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.138127 2573 generic.go:356] "Generic (PLEG): container finished" podID="520ec79b-a25c-454c-a1e1-ba9572490cfd" containerID="671c040ccd6afa57d831e6aab55cfe14f0e22a7f294cec71e590c33be8d1d5fc" exitCode=0 Sep 11 14:21:45.138227 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.138171 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" event={"ID":"520ec79b-a25c-454c-a1e1-ba9572490cfd","Type":"ContainerDied","Data":"671c040ccd6afa57d831e6aab55cfe14f0e22a7f294cec71e590c33be8d1d5fc"} Sep 11 14:21:45.270868 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.270848 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" Sep 11 14:21:45.430800 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.430719 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f401b3e-099a-414d-bd11-a8ae40d61464-bundle\") pod \"8f401b3e-099a-414d-bd11-a8ae40d61464\" (UID: \"8f401b3e-099a-414d-bd11-a8ae40d61464\") " Sep 11 14:21:45.430800 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.430762 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zttks\" (UniqueName: \"kubernetes.io/projected/8f401b3e-099a-414d-bd11-a8ae40d61464-kube-api-access-zttks\") pod \"8f401b3e-099a-414d-bd11-a8ae40d61464\" (UID: \"8f401b3e-099a-414d-bd11-a8ae40d61464\") " Sep 11 14:21:45.430800 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.430801 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f401b3e-099a-414d-bd11-a8ae40d61464-util\") pod \"8f401b3e-099a-414d-bd11-a8ae40d61464\" (UID: \"8f401b3e-099a-414d-bd11-a8ae40d61464\") " Sep 11 14:21:45.431217 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.431192 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f401b3e-099a-414d-bd11-a8ae40d61464-bundle" (OuterVolumeSpecName: "bundle") pod "8f401b3e-099a-414d-bd11-a8ae40d61464" (UID: "8f401b3e-099a-414d-bd11-a8ae40d61464"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:21:45.432910 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.432882 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f401b3e-099a-414d-bd11-a8ae40d61464-kube-api-access-zttks" (OuterVolumeSpecName: "kube-api-access-zttks") pod "8f401b3e-099a-414d-bd11-a8ae40d61464" (UID: "8f401b3e-099a-414d-bd11-a8ae40d61464"). InnerVolumeSpecName "kube-api-access-zttks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:21:45.438982 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.438954 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f401b3e-099a-414d-bd11-a8ae40d61464-util" (OuterVolumeSpecName: "util") pod "8f401b3e-099a-414d-bd11-a8ae40d61464" (UID: "8f401b3e-099a-414d-bd11-a8ae40d61464"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:21:45.532103 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.532077 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f401b3e-099a-414d-bd11-a8ae40d61464-bundle\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:21:45.532103 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.532100 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zttks\" (UniqueName: \"kubernetes.io/projected/8f401b3e-099a-414d-bd11-a8ae40d61464-kube-api-access-zttks\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:21:45.532250 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:45.532110 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f401b3e-099a-414d-bd11-a8ae40d61464-util\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:21:46.145466 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:46.145439 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" Sep 11 14:21:46.145808 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:46.145439 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lg2ht" event={"ID":"8f401b3e-099a-414d-bd11-a8ae40d61464","Type":"ContainerDied","Data":"625465d3fae76a8c0cc143199176399b682f821d76bb11f479f9646df8bcf5aa"} Sep 11 14:21:46.145808 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:46.145564 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="625465d3fae76a8c0cc143199176399b682f821d76bb11f479f9646df8bcf5aa" Sep 11 14:21:46.281424 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:46.281401 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" Sep 11 14:21:46.439825 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:46.439742 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/520ec79b-a25c-454c-a1e1-ba9572490cfd-bundle\") pod \"520ec79b-a25c-454c-a1e1-ba9572490cfd\" (UID: \"520ec79b-a25c-454c-a1e1-ba9572490cfd\") " Sep 11 14:21:46.439825 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:46.439776 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/520ec79b-a25c-454c-a1e1-ba9572490cfd-util\") pod \"520ec79b-a25c-454c-a1e1-ba9572490cfd\" (UID: \"520ec79b-a25c-454c-a1e1-ba9572490cfd\") " Sep 11 14:21:46.439825 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:46.439816 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n9nj\" (UniqueName: \"kubernetes.io/projected/520ec79b-a25c-454c-a1e1-ba9572490cfd-kube-api-access-5n9nj\") pod \"520ec79b-a25c-454c-a1e1-ba9572490cfd\" (UID: \"520ec79b-a25c-454c-a1e1-ba9572490cfd\") " Sep 11 14:21:46.440311 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:46.440287 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520ec79b-a25c-454c-a1e1-ba9572490cfd-bundle" (OuterVolumeSpecName: "bundle") pod "520ec79b-a25c-454c-a1e1-ba9572490cfd" (UID: "520ec79b-a25c-454c-a1e1-ba9572490cfd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:21:46.441902 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:46.441881 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520ec79b-a25c-454c-a1e1-ba9572490cfd-kube-api-access-5n9nj" (OuterVolumeSpecName: "kube-api-access-5n9nj") pod "520ec79b-a25c-454c-a1e1-ba9572490cfd" (UID: "520ec79b-a25c-454c-a1e1-ba9572490cfd"). InnerVolumeSpecName "kube-api-access-5n9nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:21:46.444980 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:46.444940 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520ec79b-a25c-454c-a1e1-ba9572490cfd-util" (OuterVolumeSpecName: "util") pod "520ec79b-a25c-454c-a1e1-ba9572490cfd" (UID: "520ec79b-a25c-454c-a1e1-ba9572490cfd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:21:46.540912 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:46.540878 2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/520ec79b-a25c-454c-a1e1-ba9572490cfd-bundle\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:21:46.540912 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:46.540905 2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/520ec79b-a25c-454c-a1e1-ba9572490cfd-util\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:21:46.540912 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:46.540914 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n9nj\" (UniqueName: \"kubernetes.io/projected/520ec79b-a25c-454c-a1e1-ba9572490cfd-kube-api-access-5n9nj\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:21:47.152044 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:47.152012 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" event={"ID":"520ec79b-a25c-454c-a1e1-ba9572490cfd","Type":"ContainerDied","Data":"ad985ed0c67f311f3c11c80b37723f1d45db64e079ef0cbeb6b566a752b03d12"} Sep 11 14:21:47.152044 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:47.152044 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q5lbc" Sep 11 14:21:47.152459 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:21:47.152057 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad985ed0c67f311f3c11c80b37723f1d45db64e079ef0cbeb6b566a752b03d12" Sep 11 14:23:00.452612 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.452575 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8"] Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453026 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12" containerName="pull" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453040 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12" containerName="pull" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453050 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f401b3e-099a-414d-bd11-a8ae40d61464" containerName="pull" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453055 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f401b3e-099a-414d-bd11-a8ae40d61464" containerName="pull" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453065 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12" containerName="extract" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453070 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12" containerName="extract" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453079 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f401b3e-099a-414d-bd11-a8ae40d61464" containerName="util" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453084 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f401b3e-099a-414d-bd11-a8ae40d61464" containerName="util" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453091 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="520ec79b-a25c-454c-a1e1-ba9572490cfd" containerName="pull" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453096 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="520ec79b-a25c-454c-a1e1-ba9572490cfd" containerName="pull" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453109 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a538801-cd1b-48a9-8855-baeefff75ec7" containerName="pull" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453114 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a538801-cd1b-48a9-8855-baeefff75ec7" containerName="pull" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453121 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="520ec79b-a25c-454c-a1e1-ba9572490cfd" containerName="extract" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453126 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="520ec79b-a25c-454c-a1e1-ba9572490cfd" containerName="extract" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453134 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="520ec79b-a25c-454c-a1e1-ba9572490cfd" containerName="util" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453139 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="520ec79b-a25c-454c-a1e1-ba9572490cfd" containerName="util" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453148 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a538801-cd1b-48a9-8855-baeefff75ec7" containerName="util" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453153 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a538801-cd1b-48a9-8855-baeefff75ec7" containerName="util" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453159 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12" containerName="util" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453164 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12" containerName="util" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453169 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f401b3e-099a-414d-bd11-a8ae40d61464" containerName="extract" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453174 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f401b3e-099a-414d-bd11-a8ae40d61464" containerName="extract" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453181 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a538801-cd1b-48a9-8855-baeefff75ec7" containerName="extract" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453186 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a538801-cd1b-48a9-8855-baeefff75ec7" containerName="extract" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453242 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ad8b04d-bc8d-4fc7-ab5e-49853abdcf12" containerName="extract" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453251 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a538801-cd1b-48a9-8855-baeefff75ec7" containerName="extract" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453258 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f401b3e-099a-414d-bd11-a8ae40d61464" containerName="extract" Sep 11 14:23:00.453331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.453268 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="520ec79b-a25c-454c-a1e1-ba9572490cfd" containerName="extract" Sep 11 14:23:00.456265 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.456250 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.459710 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.459690 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8ndg6\"" Sep 11 14:23:00.483785 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.483760 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8"] Sep 11 14:23:00.550060 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.550032 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7edf42e8-89d8-4376-b88f-0693f77cbd91-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.550216 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.550067 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7edf42e8-89d8-4376-b88f-0693f77cbd91-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.550216 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.550088 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edf42e8-89d8-4376-b88f-0693f77cbd91-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.550216 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.550104 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdgz9\" (UniqueName: \"kubernetes.io/projected/7edf42e8-89d8-4376-b88f-0693f77cbd91-kube-api-access-gdgz9\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.550216 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.550132 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7edf42e8-89d8-4376-b88f-0693f77cbd91-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.550216 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.550154 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7edf42e8-89d8-4376-b88f-0693f77cbd91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.550412 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.550247 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edf42e8-89d8-4376-b88f-0693f77cbd91-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.550412 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.550277 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7edf42e8-89d8-4376-b88f-0693f77cbd91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.550412 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.550297 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7edf42e8-89d8-4376-b88f-0693f77cbd91-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.650776 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.650746 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7edf42e8-89d8-4376-b88f-0693f77cbd91-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.650958 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.650783 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edf42e8-89d8-4376-b88f-0693f77cbd91-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.650958 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.650807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdgz9\" (UniqueName: \"kubernetes.io/projected/7edf42e8-89d8-4376-b88f-0693f77cbd91-kube-api-access-gdgz9\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.650958 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.650857 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7edf42e8-89d8-4376-b88f-0693f77cbd91-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.650958 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.650892 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7edf42e8-89d8-4376-b88f-0693f77cbd91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.651177 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.650990 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edf42e8-89d8-4376-b88f-0693f77cbd91-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.651177 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.651020 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7edf42e8-89d8-4376-b88f-0693f77cbd91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.651177 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.651051 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7edf42e8-89d8-4376-b88f-0693f77cbd91-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.651177 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.651087 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7edf42e8-89d8-4376-b88f-0693f77cbd91-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.651399 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.651211 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edf42e8-89d8-4376-b88f-0693f77cbd91-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.651399 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.651288 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7edf42e8-89d8-4376-b88f-0693f77cbd91-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.651399 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.651392 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7edf42e8-89d8-4376-b88f-0693f77cbd91-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.651595 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.651480 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edf42e8-89d8-4376-b88f-0693f77cbd91-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.651688 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.651654 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7edf42e8-89d8-4376-b88f-0693f77cbd91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.653220 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.653191 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7edf42e8-89d8-4376-b88f-0693f77cbd91-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.653546 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.653526 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7edf42e8-89d8-4376-b88f-0693f77cbd91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.671275 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.671249 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdgz9\" (UniqueName: \"kubernetes.io/projected/7edf42e8-89d8-4376-b88f-0693f77cbd91-kube-api-access-gdgz9\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.671376 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.671333 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7edf42e8-89d8-4376-b88f-0693f77cbd91-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-d5dd8\" (UID: \"7edf42e8-89d8-4376-b88f-0693f77cbd91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:00.766077 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:00.766014 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:01.047896 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:01.047869 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8"] Sep 11 14:23:01.049207 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:23:01.049179 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7edf42e8_89d8_4376_b88f_0693f77cbd91.slice/crio-50858b4f1aa3a5d99b8049b06c5c1d0de71be43598a6dae5aa2511f03e9c9e0a WatchSource:0}: Error finding container 50858b4f1aa3a5d99b8049b06c5c1d0de71be43598a6dae5aa2511f03e9c9e0a: Status 404 returned error can't find the container with id 50858b4f1aa3a5d99b8049b06c5c1d0de71be43598a6dae5aa2511f03e9c9e0a Sep 11 14:23:01.051319 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:01.051288 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:23:01.051402 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:01.051353 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:23:01.051402 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:01.051379 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:23:01.538338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:01.538297 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" event={"ID":"7edf42e8-89d8-4376-b88f-0693f77cbd91","Type":"ContainerStarted","Data":"67b1eac861712439462d70322777a094991b006760a71c2f31fa57f575ac1d2a"} Sep 11 14:23:01.538803 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:01.538360 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" event={"ID":"7edf42e8-89d8-4376-b88f-0693f77cbd91","Type":"ContainerStarted","Data":"50858b4f1aa3a5d99b8049b06c5c1d0de71be43598a6dae5aa2511f03e9c9e0a"} Sep 11 14:23:01.564059 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:01.564016 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" podStartSLOduration=1.5640027079999999 podStartE2EDuration="1.564002708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:23:01.563341452 +0000 UTC m=+493.900309646" watchObservedRunningTime="2026-09-11 14:23:01.564002708 +0000 UTC m=+493.900970898" Sep 11 14:23:01.766903 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:01.766872 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:02.772395 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:02.772355 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:03.549327 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:03.549294 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:03.550258 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:03.550238 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:23:03.713953 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:03.713912 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b2rwg"] Sep 11 14:23:03.717374 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:03.717347 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b2rwg" Sep 11 14:23:03.721028 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:03.721006 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 14:23:03.724177 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:03.724156 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7rqgx\"" Sep 11 14:23:03.724510 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:03.724490 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 14:23:03.742904 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:03.742879 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b2rwg"] Sep 11 14:23:03.776904 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:03.776882 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4vpl\" (UniqueName: \"kubernetes.io/projected/050bdf2e-6b4f-4da6-ac61-4aab9919ddc8-kube-api-access-x4vpl\") pod \"authorino-7bcfd8f9cd-b2rwg\" (UID: \"050bdf2e-6b4f-4da6-ac61-4aab9919ddc8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-b2rwg" Sep 11 14:23:03.878036 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:03.878007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4vpl\" (UniqueName: \"kubernetes.io/projected/050bdf2e-6b4f-4da6-ac61-4aab9919ddc8-kube-api-access-x4vpl\") pod \"authorino-7bcfd8f9cd-b2rwg\" (UID: \"050bdf2e-6b4f-4da6-ac61-4aab9919ddc8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-b2rwg" Sep 11 14:23:03.896473 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:03.896449 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4vpl\" (UniqueName: \"kubernetes.io/projected/050bdf2e-6b4f-4da6-ac61-4aab9919ddc8-kube-api-access-x4vpl\") pod \"authorino-7bcfd8f9cd-b2rwg\" (UID: \"050bdf2e-6b4f-4da6-ac61-4aab9919ddc8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-b2rwg" Sep 11 14:23:04.027021 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:04.026994 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b2rwg" Sep 11 14:23:04.045376 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:04.045351 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-l7d86"] Sep 11 14:23:04.049377 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:04.049359 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-l7d86" Sep 11 14:23:04.064029 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:04.064008 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-l7d86"] Sep 11 14:23:04.181174 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:04.181083 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qclsw\" (UniqueName: \"kubernetes.io/projected/9d62b53c-8932-4234-be53-af7555e10dca-kube-api-access-qclsw\") pod \"authorino-59c974bd48-l7d86\" (UID: \"9d62b53c-8932-4234-be53-af7555e10dca\") " pod="kuadrant-system/authorino-59c974bd48-l7d86" Sep 11 14:23:04.281904 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:04.281872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qclsw\" (UniqueName: \"kubernetes.io/projected/9d62b53c-8932-4234-be53-af7555e10dca-kube-api-access-qclsw\") pod \"authorino-59c974bd48-l7d86\" (UID: \"9d62b53c-8932-4234-be53-af7555e10dca\") " pod="kuadrant-system/authorino-59c974bd48-l7d86" Sep 11 14:23:04.293569 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:04.293531 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qclsw\" (UniqueName: \"kubernetes.io/projected/9d62b53c-8932-4234-be53-af7555e10dca-kube-api-access-qclsw\") pod \"authorino-59c974bd48-l7d86\" (UID: \"9d62b53c-8932-4234-be53-af7555e10dca\") " pod="kuadrant-system/authorino-59c974bd48-l7d86" Sep 11 14:23:04.319170 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:04.319150 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b2rwg"] Sep 11 14:23:04.321208 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:23:04.321176 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod050bdf2e_6b4f_4da6_ac61_4aab9919ddc8.slice/crio-11cffd01b0ab05c1f577289b342f585233fd283c28b95360efcf1ec806a2cbe3 WatchSource:0}: Error finding container 11cffd01b0ab05c1f577289b342f585233fd283c28b95360efcf1ec806a2cbe3: Status 404 returned error can't find the container with id 11cffd01b0ab05c1f577289b342f585233fd283c28b95360efcf1ec806a2cbe3 Sep 11 14:23:04.361247 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:04.361227 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-l7d86" Sep 11 14:23:04.557564 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:04.557471 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b2rwg" event={"ID":"050bdf2e-6b4f-4da6-ac61-4aab9919ddc8","Type":"ContainerStarted","Data":"11cffd01b0ab05c1f577289b342f585233fd283c28b95360efcf1ec806a2cbe3"} Sep 11 14:23:04.635511 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:04.635484 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-l7d86"] Sep 11 14:23:04.636188 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:23:04.636168 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d62b53c_8932_4234_be53_af7555e10dca.slice/crio-b84f568aa8177b000b391ec9d76fb042fd4e507cf47354310d9f290eb3a5ff77 WatchSource:0}: Error finding container b84f568aa8177b000b391ec9d76fb042fd4e507cf47354310d9f290eb3a5ff77: Status 404 returned error can't find the container with id b84f568aa8177b000b391ec9d76fb042fd4e507cf47354310d9f290eb3a5ff77 Sep 11 14:23:05.569174 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:05.569106 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-l7d86" event={"ID":"9d62b53c-8932-4234-be53-af7555e10dca","Type":"ContainerStarted","Data":"b84f568aa8177b000b391ec9d76fb042fd4e507cf47354310d9f290eb3a5ff77"} Sep 11 14:23:07.583133 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:07.583098 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b2rwg" event={"ID":"050bdf2e-6b4f-4da6-ac61-4aab9919ddc8","Type":"ContainerStarted","Data":"d3501b0e6ca73292c82873e04ce2c4c4a88bdc3b69be4f0c525ef1c29581cc8a"} Sep 11 14:23:07.584705 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:07.584678 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-l7d86" event={"ID":"9d62b53c-8932-4234-be53-af7555e10dca","Type":"ContainerStarted","Data":"3e9e63af8732c2cce8ee3f07b9bbce3ee6b2628821538e48a52384b743b1b66c"} Sep 11 14:23:07.605228 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:07.605180 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-b2rwg" podStartSLOduration=1.9814900199999999 podStartE2EDuration="4.605164538s" totalImagesPullingTime="2.623674518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:23:07.60244793 +0000 UTC m=+499.939416120" watchObservedRunningTime="2026-09-11 14:23:07.605164538 +0000 UTC m=+499.942132729" Sep 11 14:23:07.629955 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:07.629904 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-l7d86" podStartSLOduration=1.309566263 podStartE2EDuration="3.629888802s" totalImagesPullingTime="2.320322539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:23:07.628382476 +0000 UTC m=+499.965350678" watchObservedRunningTime="2026-09-11 14:23:07.629888802 +0000 UTC m=+499.966856993" Sep 11 14:23:07.669331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:07.669297 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b2rwg"] Sep 11 14:23:09.595868 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:09.595795 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-b2rwg" podUID="050bdf2e-6b4f-4da6-ac61-4aab9919ddc8" containerName="authorino" containerID="cri-o://d3501b0e6ca73292c82873e04ce2c4c4a88bdc3b69be4f0c525ef1c29581cc8a" gracePeriod=30 Sep 11 14:23:09.832523 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:09.832500 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b2rwg" Sep 11 14:23:09.930504 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:09.930420 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4vpl\" (UniqueName: \"kubernetes.io/projected/050bdf2e-6b4f-4da6-ac61-4aab9919ddc8-kube-api-access-x4vpl\") pod \"050bdf2e-6b4f-4da6-ac61-4aab9919ddc8\" (UID: \"050bdf2e-6b4f-4da6-ac61-4aab9919ddc8\") " Sep 11 14:23:09.932607 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:09.932582 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050bdf2e-6b4f-4da6-ac61-4aab9919ddc8-kube-api-access-x4vpl" (OuterVolumeSpecName: "kube-api-access-x4vpl") pod "050bdf2e-6b4f-4da6-ac61-4aab9919ddc8" (UID: "050bdf2e-6b4f-4da6-ac61-4aab9919ddc8"). InnerVolumeSpecName "kube-api-access-x4vpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:10.031301 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:10.031267 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4vpl\" (UniqueName: \"kubernetes.io/projected/050bdf2e-6b4f-4da6-ac61-4aab9919ddc8-kube-api-access-x4vpl\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:23:10.602059 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:10.602031 2573 generic.go:356] "Generic (PLEG): container finished" podID="050bdf2e-6b4f-4da6-ac61-4aab9919ddc8" containerID="d3501b0e6ca73292c82873e04ce2c4c4a88bdc3b69be4f0c525ef1c29581cc8a" exitCode=0 Sep 11 14:23:10.602419 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:10.602069 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b2rwg" Sep 11 14:23:10.602419 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:10.602115 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b2rwg" event={"ID":"050bdf2e-6b4f-4da6-ac61-4aab9919ddc8","Type":"ContainerDied","Data":"d3501b0e6ca73292c82873e04ce2c4c4a88bdc3b69be4f0c525ef1c29581cc8a"} Sep 11 14:23:10.602419 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:10.602153 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b2rwg" event={"ID":"050bdf2e-6b4f-4da6-ac61-4aab9919ddc8","Type":"ContainerDied","Data":"11cffd01b0ab05c1f577289b342f585233fd283c28b95360efcf1ec806a2cbe3"} Sep 11 14:23:10.602419 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:10.602171 2573 scope.go:117] "RemoveContainer" containerID="d3501b0e6ca73292c82873e04ce2c4c4a88bdc3b69be4f0c525ef1c29581cc8a" Sep 11 14:23:10.611085 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:10.611065 2573 scope.go:117] "RemoveContainer" containerID="d3501b0e6ca73292c82873e04ce2c4c4a88bdc3b69be4f0c525ef1c29581cc8a" Sep 11 14:23:10.611345 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:23:10.611328 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3501b0e6ca73292c82873e04ce2c4c4a88bdc3b69be4f0c525ef1c29581cc8a\": container with ID starting with d3501b0e6ca73292c82873e04ce2c4c4a88bdc3b69be4f0c525ef1c29581cc8a not found: ID does not exist" containerID="d3501b0e6ca73292c82873e04ce2c4c4a88bdc3b69be4f0c525ef1c29581cc8a" Sep 11 14:23:10.611388 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:10.611354 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3501b0e6ca73292c82873e04ce2c4c4a88bdc3b69be4f0c525ef1c29581cc8a"} err="failed to get container status \"d3501b0e6ca73292c82873e04ce2c4c4a88bdc3b69be4f0c525ef1c29581cc8a\": rpc error: code = NotFound desc = could not find container \"d3501b0e6ca73292c82873e04ce2c4c4a88bdc3b69be4f0c525ef1c29581cc8a\": container with ID starting with d3501b0e6ca73292c82873e04ce2c4c4a88bdc3b69be4f0c525ef1c29581cc8a not found: ID does not exist" Sep 11 14:23:10.625705 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:10.625680 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b2rwg"] Sep 11 14:23:10.632477 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:10.632456 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b2rwg"] Sep 11 14:23:12.186785 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:12.186752 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="050bdf2e-6b4f-4da6-ac61-4aab9919ddc8" path="/var/lib/kubelet/pods/050bdf2e-6b4f-4da6-ac61-4aab9919ddc8/volumes" Sep 11 14:23:31.185555 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.185522 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-687g5"] Sep 11 14:23:31.185944 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.185929 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050bdf2e-6b4f-4da6-ac61-4aab9919ddc8" containerName="authorino" Sep 11 14:23:31.185944 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.185942 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="050bdf2e-6b4f-4da6-ac61-4aab9919ddc8" containerName="authorino" Sep 11 14:23:31.186028 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.186017 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="050bdf2e-6b4f-4da6-ac61-4aab9919ddc8" containerName="authorino" Sep 11 14:23:31.189067 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.189051 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-687g5" Sep 11 14:23:31.207231 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.207202 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-687g5"] Sep 11 14:23:31.317027 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.316996 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjxlj\" (UniqueName: \"kubernetes.io/projected/0c081769-a685-4ed7-bd60-3dafd7455813-kube-api-access-fjxlj\") pod \"authorino-557f876f69-687g5\" (UID: \"0c081769-a685-4ed7-bd60-3dafd7455813\") " pod="kuadrant-system/authorino-557f876f69-687g5" Sep 11 14:23:31.333322 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.333291 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-687g5"] Sep 11 14:23:31.333563 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:23:31.333546 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fjxlj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-687g5" podUID="0c081769-a685-4ed7-bd60-3dafd7455813" Sep 11 14:23:31.375828 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.375799 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6sjlp"] Sep 11 14:23:31.379107 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.379091 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-6sjlp" Sep 11 14:23:31.383535 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.383516 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 11 14:23:31.394861 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.394841 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6sjlp"] Sep 11 14:23:31.418237 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.418213 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjxlj\" (UniqueName: \"kubernetes.io/projected/0c081769-a685-4ed7-bd60-3dafd7455813-kube-api-access-fjxlj\") pod \"authorino-557f876f69-687g5\" (UID: \"0c081769-a685-4ed7-bd60-3dafd7455813\") " pod="kuadrant-system/authorino-557f876f69-687g5" Sep 11 14:23:31.431723 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.431697 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjxlj\" (UniqueName: \"kubernetes.io/projected/0c081769-a685-4ed7-bd60-3dafd7455813-kube-api-access-fjxlj\") pod \"authorino-557f876f69-687g5\" (UID: \"0c081769-a685-4ed7-bd60-3dafd7455813\") " pod="kuadrant-system/authorino-557f876f69-687g5" Sep 11 14:23:31.478272 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.478205 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6sjlp"] Sep 11 14:23:31.478496 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:23:31.478474 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dz286 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-6sjlp" podUID="f58ae779-886a-48eb-b045-cb8118cf0f04" Sep 11 14:23:31.518822 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.518779 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz286\" (UniqueName: \"kubernetes.io/projected/f58ae779-886a-48eb-b045-cb8118cf0f04-kube-api-access-dz286\") pod \"authorino-6d4dfc649f-6sjlp\" (UID: \"f58ae779-886a-48eb-b045-cb8118cf0f04\") " pod="kuadrant-system/authorino-6d4dfc649f-6sjlp" Sep 11 14:23:31.518949 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.518928 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f58ae779-886a-48eb-b045-cb8118cf0f04-tls-cert\") pod \"authorino-6d4dfc649f-6sjlp\" (UID: \"f58ae779-886a-48eb-b045-cb8118cf0f04\") " pod="kuadrant-system/authorino-6d4dfc649f-6sjlp" Sep 11 14:23:31.521084 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.521065 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84cdbd95d5-mcbxc"] Sep 11 14:23:31.524409 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.524393 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84cdbd95d5-mcbxc" Sep 11 14:23:31.539640 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.539376 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84cdbd95d5-mcbxc"] Sep 11 14:23:31.619930 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.619896 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvvdk\" (UniqueName: \"kubernetes.io/projected/4000e4a9-2645-49c1-a9f1-3e91dfdb333a-kube-api-access-bvvdk\") pod \"authorino-84cdbd95d5-mcbxc\" (UID: \"4000e4a9-2645-49c1-a9f1-3e91dfdb333a\") " pod="kuadrant-system/authorino-84cdbd95d5-mcbxc" Sep 11 14:23:31.620167 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.619947 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz286\" (UniqueName: \"kubernetes.io/projected/f58ae779-886a-48eb-b045-cb8118cf0f04-kube-api-access-dz286\") pod \"authorino-6d4dfc649f-6sjlp\" (UID: \"f58ae779-886a-48eb-b045-cb8118cf0f04\") " pod="kuadrant-system/authorino-6d4dfc649f-6sjlp" Sep 11 14:23:31.620167 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.620056 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4000e4a9-2645-49c1-a9f1-3e91dfdb333a-tls-cert\") pod \"authorino-84cdbd95d5-mcbxc\" (UID: \"4000e4a9-2645-49c1-a9f1-3e91dfdb333a\") " pod="kuadrant-system/authorino-84cdbd95d5-mcbxc" Sep 11 14:23:31.620167 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.620086 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f58ae779-886a-48eb-b045-cb8118cf0f04-tls-cert\") pod \"authorino-6d4dfc649f-6sjlp\" (UID: \"f58ae779-886a-48eb-b045-cb8118cf0f04\") " pod="kuadrant-system/authorino-6d4dfc649f-6sjlp" Sep 11 14:23:31.622401 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.622375 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f58ae779-886a-48eb-b045-cb8118cf0f04-tls-cert\") pod \"authorino-6d4dfc649f-6sjlp\" (UID: \"f58ae779-886a-48eb-b045-cb8118cf0f04\") " pod="kuadrant-system/authorino-6d4dfc649f-6sjlp" Sep 11 14:23:31.628896 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.628875 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz286\" (UniqueName: \"kubernetes.io/projected/f58ae779-886a-48eb-b045-cb8118cf0f04-kube-api-access-dz286\") pod \"authorino-6d4dfc649f-6sjlp\" (UID: \"f58ae779-886a-48eb-b045-cb8118cf0f04\") " pod="kuadrant-system/authorino-6d4dfc649f-6sjlp" Sep 11 14:23:31.718334 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.718302 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-6sjlp" Sep 11 14:23:31.718511 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.718302 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-687g5" Sep 11 14:23:31.720781 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.720755 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4000e4a9-2645-49c1-a9f1-3e91dfdb333a-tls-cert\") pod \"authorino-84cdbd95d5-mcbxc\" (UID: \"4000e4a9-2645-49c1-a9f1-3e91dfdb333a\") " pod="kuadrant-system/authorino-84cdbd95d5-mcbxc" Sep 11 14:23:31.720978 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.720854 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvvdk\" (UniqueName: \"kubernetes.io/projected/4000e4a9-2645-49c1-a9f1-3e91dfdb333a-kube-api-access-bvvdk\") pod \"authorino-84cdbd95d5-mcbxc\" (UID: \"4000e4a9-2645-49c1-a9f1-3e91dfdb333a\") " pod="kuadrant-system/authorino-84cdbd95d5-mcbxc" Sep 11 14:23:31.723203 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.723183 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4000e4a9-2645-49c1-a9f1-3e91dfdb333a-tls-cert\") pod \"authorino-84cdbd95d5-mcbxc\" (UID: \"4000e4a9-2645-49c1-a9f1-3e91dfdb333a\") " pod="kuadrant-system/authorino-84cdbd95d5-mcbxc" Sep 11 14:23:31.725850 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.725817 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-6sjlp" Sep 11 14:23:31.735383 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.735322 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvvdk\" (UniqueName: \"kubernetes.io/projected/4000e4a9-2645-49c1-a9f1-3e91dfdb333a-kube-api-access-bvvdk\") pod \"authorino-84cdbd95d5-mcbxc\" (UID: \"4000e4a9-2645-49c1-a9f1-3e91dfdb333a\") " pod="kuadrant-system/authorino-84cdbd95d5-mcbxc" Sep 11 14:23:31.737006 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.736988 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-687g5" Sep 11 14:23:31.822193 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.822164 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz286\" (UniqueName: \"kubernetes.io/projected/f58ae779-886a-48eb-b045-cb8118cf0f04-kube-api-access-dz286\") pod \"f58ae779-886a-48eb-b045-cb8118cf0f04\" (UID: \"f58ae779-886a-48eb-b045-cb8118cf0f04\") " Sep 11 14:23:31.822370 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.822223 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjxlj\" (UniqueName: \"kubernetes.io/projected/0c081769-a685-4ed7-bd60-3dafd7455813-kube-api-access-fjxlj\") pod \"0c081769-a685-4ed7-bd60-3dafd7455813\" (UID: \"0c081769-a685-4ed7-bd60-3dafd7455813\") " Sep 11 14:23:31.822370 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.822260 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f58ae779-886a-48eb-b045-cb8118cf0f04-tls-cert\") pod \"f58ae779-886a-48eb-b045-cb8118cf0f04\" (UID: \"f58ae779-886a-48eb-b045-cb8118cf0f04\") " Sep 11 14:23:31.824277 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.824245 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58ae779-886a-48eb-b045-cb8118cf0f04-kube-api-access-dz286" (OuterVolumeSpecName: "kube-api-access-dz286") pod "f58ae779-886a-48eb-b045-cb8118cf0f04" (UID: "f58ae779-886a-48eb-b045-cb8118cf0f04"). InnerVolumeSpecName "kube-api-access-dz286". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:31.824277 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.824269 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c081769-a685-4ed7-bd60-3dafd7455813-kube-api-access-fjxlj" (OuterVolumeSpecName: "kube-api-access-fjxlj") pod "0c081769-a685-4ed7-bd60-3dafd7455813" (UID: "0c081769-a685-4ed7-bd60-3dafd7455813"). InnerVolumeSpecName "kube-api-access-fjxlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:31.824414 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.824292 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58ae779-886a-48eb-b045-cb8118cf0f04-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f58ae779-886a-48eb-b045-cb8118cf0f04" (UID: "f58ae779-886a-48eb-b045-cb8118cf0f04"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:23:31.833451 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.833433 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84cdbd95d5-mcbxc" Sep 11 14:23:31.923171 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.923144 2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f58ae779-886a-48eb-b045-cb8118cf0f04-tls-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:23:31.923171 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.923170 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz286\" (UniqueName: \"kubernetes.io/projected/f58ae779-886a-48eb-b045-cb8118cf0f04-kube-api-access-dz286\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:23:31.923328 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:31.923180 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjxlj\" (UniqueName: \"kubernetes.io/projected/0c081769-a685-4ed7-bd60-3dafd7455813-kube-api-access-fjxlj\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:23:32.115626 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:32.114748 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84cdbd95d5-mcbxc"] Sep 11 14:23:32.117915 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:23:32.117886 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4000e4a9_2645_49c1_a9f1_3e91dfdb333a.slice/crio-f26a567b34a1d1707cf06958fee3e5d218485b271c8a6565884fc34d67ffd7ec WatchSource:0}: Error finding container f26a567b34a1d1707cf06958fee3e5d218485b271c8a6565884fc34d67ffd7ec: Status 404 returned error can't find the container with id f26a567b34a1d1707cf06958fee3e5d218485b271c8a6565884fc34d67ffd7ec Sep 11 14:23:32.724933 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:32.724858 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-687g5" Sep 11 14:23:32.724933 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:32.724898 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-6sjlp" Sep 11 14:23:32.725403 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:32.724850 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84cdbd95d5-mcbxc" event={"ID":"4000e4a9-2645-49c1-a9f1-3e91dfdb333a","Type":"ContainerStarted","Data":"f26a567b34a1d1707cf06958fee3e5d218485b271c8a6565884fc34d67ffd7ec"} Sep 11 14:23:32.759697 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:32.759665 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-687g5"] Sep 11 14:23:32.765984 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:32.765959 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-687g5"] Sep 11 14:23:32.800517 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:32.800487 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6sjlp"] Sep 11 14:23:32.802575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:32.802552 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6sjlp"] Sep 11 14:23:33.732057 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:33.732019 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84cdbd95d5-mcbxc" event={"ID":"4000e4a9-2645-49c1-a9f1-3e91dfdb333a","Type":"ContainerStarted","Data":"cdfe8bcbf0b3e409ad237a2d4760d6e07aa96d6bf114156592fa324e7ee761ca"} Sep 11 14:23:33.758991 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:33.758945 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84cdbd95d5-mcbxc" podStartSLOduration=1.9336180120000002 podStartE2EDuration="2.758930859s" totalImagesPullingTime="825.312847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:23:33.756843545 +0000 UTC m=+526.093811728" watchObservedRunningTime="2026-09-11 14:23:33.758930859 +0000 UTC m=+526.095899108" Sep 11 14:23:33.804478 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:33.804449 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-l7d86"] Sep 11 14:23:33.804633 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:33.804608 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-l7d86" podUID="9d62b53c-8932-4234-be53-af7555e10dca" containerName="authorino" containerID="cri-o://3e9e63af8732c2cce8ee3f07b9bbce3ee6b2628821538e48a52384b743b1b66c" gracePeriod=30 Sep 11 14:23:34.045182 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:34.045159 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-l7d86" Sep 11 14:23:34.142631 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:34.142599 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qclsw\" (UniqueName: \"kubernetes.io/projected/9d62b53c-8932-4234-be53-af7555e10dca-kube-api-access-qclsw\") pod \"9d62b53c-8932-4234-be53-af7555e10dca\" (UID: \"9d62b53c-8932-4234-be53-af7555e10dca\") " Sep 11 14:23:34.144589 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:34.144562 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d62b53c-8932-4234-be53-af7555e10dca-kube-api-access-qclsw" (OuterVolumeSpecName: "kube-api-access-qclsw") pod "9d62b53c-8932-4234-be53-af7555e10dca" (UID: "9d62b53c-8932-4234-be53-af7555e10dca"). InnerVolumeSpecName "kube-api-access-qclsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:34.187172 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:34.187141 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c081769-a685-4ed7-bd60-3dafd7455813" path="/var/lib/kubelet/pods/0c081769-a685-4ed7-bd60-3dafd7455813/volumes" Sep 11 14:23:34.187404 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:34.187391 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f58ae779-886a-48eb-b045-cb8118cf0f04" path="/var/lib/kubelet/pods/f58ae779-886a-48eb-b045-cb8118cf0f04/volumes" Sep 11 14:23:34.243634 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:34.243607 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qclsw\" (UniqueName: \"kubernetes.io/projected/9d62b53c-8932-4234-be53-af7555e10dca-kube-api-access-qclsw\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:23:34.738757 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:34.738725 2573 generic.go:356] "Generic (PLEG): container finished" podID="9d62b53c-8932-4234-be53-af7555e10dca" containerID="3e9e63af8732c2cce8ee3f07b9bbce3ee6b2628821538e48a52384b743b1b66c" exitCode=0 Sep 11 14:23:34.739186 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:34.738770 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-l7d86" Sep 11 14:23:34.739186 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:34.738817 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-l7d86" event={"ID":"9d62b53c-8932-4234-be53-af7555e10dca","Type":"ContainerDied","Data":"3e9e63af8732c2cce8ee3f07b9bbce3ee6b2628821538e48a52384b743b1b66c"} Sep 11 14:23:34.739186 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:34.738873 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-l7d86" event={"ID":"9d62b53c-8932-4234-be53-af7555e10dca","Type":"ContainerDied","Data":"b84f568aa8177b000b391ec9d76fb042fd4e507cf47354310d9f290eb3a5ff77"} Sep 11 14:23:34.739186 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:34.738891 2573 scope.go:117] "RemoveContainer" containerID="3e9e63af8732c2cce8ee3f07b9bbce3ee6b2628821538e48a52384b743b1b66c" Sep 11 14:23:34.747749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:34.747730 2573 scope.go:117] "RemoveContainer" containerID="3e9e63af8732c2cce8ee3f07b9bbce3ee6b2628821538e48a52384b743b1b66c" Sep 11 14:23:34.748172 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:23:34.748152 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9e63af8732c2cce8ee3f07b9bbce3ee6b2628821538e48a52384b743b1b66c\": container with ID starting with 3e9e63af8732c2cce8ee3f07b9bbce3ee6b2628821538e48a52384b743b1b66c not found: ID does not exist" containerID="3e9e63af8732c2cce8ee3f07b9bbce3ee6b2628821538e48a52384b743b1b66c" Sep 11 14:23:34.748216 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:34.748179 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9e63af8732c2cce8ee3f07b9bbce3ee6b2628821538e48a52384b743b1b66c"} err="failed to get container status \"3e9e63af8732c2cce8ee3f07b9bbce3ee6b2628821538e48a52384b743b1b66c\": rpc error: code = NotFound desc = could not find container \"3e9e63af8732c2cce8ee3f07b9bbce3ee6b2628821538e48a52384b743b1b66c\": container with ID starting with 3e9e63af8732c2cce8ee3f07b9bbce3ee6b2628821538e48a52384b743b1b66c not found: ID does not exist" Sep 11 14:23:34.775476 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:34.775450 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-l7d86"] Sep 11 14:23:34.786694 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:34.786670 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-l7d86"] Sep 11 14:23:36.187997 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:23:36.187963 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d62b53c-8932-4234-be53-af7555e10dca" path="/var/lib/kubelet/pods/9d62b53c-8932-4234-be53-af7555e10dca/volumes" Sep 11 14:24:28.020722 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:28.020636 2573 generic.go:356] "Generic (PLEG): container finished" podID="7edf42e8-89d8-4376-b88f-0693f77cbd91" containerID="67b1eac861712439462d70322777a094991b006760a71c2f31fa57f575ac1d2a" exitCode=0 Sep 11 14:24:28.021230 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:28.020714 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" event={"ID":"7edf42e8-89d8-4376-b88f-0693f77cbd91","Type":"ContainerDied","Data":"67b1eac861712439462d70322777a094991b006760a71c2f31fa57f575ac1d2a"} Sep 11 14:24:28.021230 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:28.021207 2573 scope.go:117] "RemoveContainer" containerID="67b1eac861712439462d70322777a094991b006760a71c2f31fa57f575ac1d2a" Sep 11 14:24:28.022257 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:28.022230 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:24:28.022352 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:28.022296 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:24:28.022352 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:28.022328 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:24:29.029734 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:29.029685 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" event={"ID":"7edf42e8-89d8-4376-b88f-0693f77cbd91","Type":"ContainerStarted","Data":"063fba42f1696f4f77ac6730ea29f04bd41603a8591147cc86f9304c5eb7aebc"} Sep 11 14:24:29.766263 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:29.766223 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:24:29.767420 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:29.767385 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d5dd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 11 14:24:29.767559 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:29.767450 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" podUID="7edf42e8-89d8-4376-b88f-0693f77cbd91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 11 14:24:30.766561 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:30.766529 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d5dd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 11 14:24:30.766965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:30.766590 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" podUID="7edf42e8-89d8-4376-b88f-0693f77cbd91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 11 14:24:31.767188 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:31.767152 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d5dd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 11 14:24:31.767601 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:31.767210 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" podUID="7edf42e8-89d8-4376-b88f-0693f77cbd91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 11 14:24:32.766672 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:32.766638 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d5dd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 11 14:24:32.766870 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:32.766689 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" podUID="7edf42e8-89d8-4376-b88f-0693f77cbd91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 11 14:24:33.549439 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:33.549397 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:24:33.766669 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:33.766636 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-d5dd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Sep 11 14:24:33.766858 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:33.766688 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" podUID="7edf42e8-89d8-4376-b88f-0693f77cbd91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Sep 11 14:24:34.770613 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:34.770581 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:24:34.771504 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:34.771481 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-d5dd8" Sep 11 14:24:48.525621 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.525589 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct"] Sep 11 14:24:48.525995 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.525982 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d62b53c-8932-4234-be53-af7555e10dca" containerName="authorino" Sep 11 14:24:48.525995 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.525994 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d62b53c-8932-4234-be53-af7555e10dca" containerName="authorino" Sep 11 14:24:48.526067 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.526050 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d62b53c-8932-4234-be53-af7555e10dca" containerName="authorino" Sep 11 14:24:48.528941 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.528923 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.535567 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.535343 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 14:24:48.535567 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.535355 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 14:24:48.536407 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.536387 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 11 14:24:48.541408 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.541392 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jg47g\"" Sep 11 14:24:48.550599 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.550574 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct"] Sep 11 14:24:48.689343 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.689313 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41c465b-64ea-4726-8b6a-a7247c636199-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.689515 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.689358 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41c465b-64ea-4726-8b6a-a7247c636199-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.689515 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.689421 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41c465b-64ea-4726-8b6a-a7247c636199-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.689515 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.689478 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41c465b-64ea-4726-8b6a-a7247c636199-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.689515 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.689496 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfqhg\" (UniqueName: \"kubernetes.io/projected/e41c465b-64ea-4726-8b6a-a7247c636199-kube-api-access-bfqhg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.689671 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.689563 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41c465b-64ea-4726-8b6a-a7247c636199-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.790779 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.790705 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41c465b-64ea-4726-8b6a-a7247c636199-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.790779 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.790755 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41c465b-64ea-4726-8b6a-a7247c636199-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.791032 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.790791 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41c465b-64ea-4726-8b6a-a7247c636199-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.791032 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.790825 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41c465b-64ea-4726-8b6a-a7247c636199-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.791032 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.790940 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41c465b-64ea-4726-8b6a-a7247c636199-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.791032 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.790983 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfqhg\" (UniqueName: \"kubernetes.io/projected/e41c465b-64ea-4726-8b6a-a7247c636199-kube-api-access-bfqhg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.791233 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.791115 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41c465b-64ea-4726-8b6a-a7247c636199-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.791233 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.791190 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41c465b-64ea-4726-8b6a-a7247c636199-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.791334 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.791227 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41c465b-64ea-4726-8b6a-a7247c636199-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.793008 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.792985 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41c465b-64ea-4726-8b6a-a7247c636199-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.793438 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.793418 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41c465b-64ea-4726-8b6a-a7247c636199-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.802041 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.802022 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfqhg\" (UniqueName: \"kubernetes.io/projected/e41c465b-64ea-4726-8b6a-a7247c636199-kube-api-access-bfqhg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct\" (UID: \"e41c465b-64ea-4726-8b6a-a7247c636199\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:48.839049 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:48.839024 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:49.126072 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:49.126041 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct"] Sep 11 14:24:49.127787 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:24:49.127761 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41c465b_64ea_4726_8b6a_a7247c636199.slice/crio-cdc0b5d8555dc6d743711c0688983547c18e350b402bf423f795b9b37720bbb2 WatchSource:0}: Error finding container cdc0b5d8555dc6d743711c0688983547c18e350b402bf423f795b9b37720bbb2: Status 404 returned error can't find the container with id cdc0b5d8555dc6d743711c0688983547c18e350b402bf423f795b9b37720bbb2 Sep 11 14:24:49.146956 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:49.146925 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" event={"ID":"e41c465b-64ea-4726-8b6a-a7247c636199","Type":"ContainerStarted","Data":"cdc0b5d8555dc6d743711c0688983547c18e350b402bf423f795b9b37720bbb2"} Sep 11 14:24:51.160688 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:51.160648 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" event={"ID":"e41c465b-64ea-4726-8b6a-a7247c636199","Type":"ContainerStarted","Data":"307e2bbeaebe2371b2617e4c4d95f381233e3e3805b10dfac4a47bdaa72325c4"} Sep 11 14:24:51.186911 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:51.186861 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" podStartSLOduration=2.030278322 podStartE2EDuration="3.186844086s" totalImagesPullingTime="1.156565764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:24:51.184016914 +0000 UTC m=+603.520985106" watchObservedRunningTime="2026-09-11 14:24:51.186844086 +0000 UTC m=+603.523812329" Sep 11 14:24:58.839880 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:58.839811 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:58.840439 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:58.839911 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:58.858074 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:58.858047 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:24:59.214754 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:24:59.214674 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct" Sep 11 14:25:28.794138 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:28.794087 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9"] Sep 11 14:25:28.799154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:28.799129 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:28.802119 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:28.802096 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 11 14:25:28.811202 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:28.811181 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9"] Sep 11 14:25:28.944593 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:28.944560 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/979f4f9c-7f7f-4f6f-932a-c258b3a09241-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:28.944593 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:28.944594 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/979f4f9c-7f7f-4f6f-932a-c258b3a09241-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:28.944897 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:28.944615 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/979f4f9c-7f7f-4f6f-932a-c258b3a09241-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:28.944897 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:28.944630 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/979f4f9c-7f7f-4f6f-932a-c258b3a09241-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:28.944897 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:28.944722 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjcld\" (UniqueName: \"kubernetes.io/projected/979f4f9c-7f7f-4f6f-932a-c258b3a09241-kube-api-access-kjcld\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:28.944897 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:28.944765 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/979f4f9c-7f7f-4f6f-932a-c258b3a09241-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:29.045546 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:29.045461 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/979f4f9c-7f7f-4f6f-932a-c258b3a09241-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:29.045546 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:29.045499 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/979f4f9c-7f7f-4f6f-932a-c258b3a09241-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:29.045546 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:29.045526 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/979f4f9c-7f7f-4f6f-932a-c258b3a09241-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:29.045546 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:29.045543 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/979f4f9c-7f7f-4f6f-932a-c258b3a09241-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:29.046079 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:29.045668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjcld\" (UniqueName: \"kubernetes.io/projected/979f4f9c-7f7f-4f6f-932a-c258b3a09241-kube-api-access-kjcld\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:29.046079 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:29.045720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/979f4f9c-7f7f-4f6f-932a-c258b3a09241-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:29.046079 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:29.045934 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/979f4f9c-7f7f-4f6f-932a-c258b3a09241-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:29.046079 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:29.046000 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/979f4f9c-7f7f-4f6f-932a-c258b3a09241-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:29.046079 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:29.046029 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/979f4f9c-7f7f-4f6f-932a-c258b3a09241-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:29.047872 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:29.047848 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/979f4f9c-7f7f-4f6f-932a-c258b3a09241-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:29.048298 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:29.048280 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/979f4f9c-7f7f-4f6f-932a-c258b3a09241-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:29.054984 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:29.054965 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjcld\" (UniqueName: \"kubernetes.io/projected/979f4f9c-7f7f-4f6f-932a-c258b3a09241-kube-api-access-kjcld\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f4sp9\" (UID: \"979f4f9c-7f7f-4f6f-932a-c258b3a09241\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:29.109145 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:29.109118 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:29.388945 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:29.388918 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9"] Sep 11 14:25:29.391074 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:25:29.391046 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod979f4f9c_7f7f_4f6f_932a_c258b3a09241.slice/crio-e184d2c300b72db686fd00c8a26a8a71e90bb71e23d2e89c8574740fea55709a WatchSource:0}: Error finding container e184d2c300b72db686fd00c8a26a8a71e90bb71e23d2e89c8574740fea55709a: Status 404 returned error can't find the container with id e184d2c300b72db686fd00c8a26a8a71e90bb71e23d2e89c8574740fea55709a Sep 11 14:25:29.392453 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:29.392431 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:25:30.373170 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:30.373127 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" event={"ID":"979f4f9c-7f7f-4f6f-932a-c258b3a09241","Type":"ContainerStarted","Data":"036ef1676925c3c6b0af1c06806273acd48e7f48be66448a3c2774544c5e605d"} Sep 11 14:25:30.373540 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:30.373188 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" event={"ID":"979f4f9c-7f7f-4f6f-932a-c258b3a09241","Type":"ContainerStarted","Data":"e184d2c300b72db686fd00c8a26a8a71e90bb71e23d2e89c8574740fea55709a"} Sep 11 14:25:30.398813 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:30.398762 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" podStartSLOduration=2.176363561 podStartE2EDuration="2.398745195s" totalImagesPullingTime="222.381634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:25:30.395264575 +0000 UTC m=+642.732232768" watchObservedRunningTime="2026-09-11 14:25:30.398745195 +0000 UTC m=+642.735713415" Sep 11 14:25:32.392575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.392540 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h"] Sep 11 14:25:32.396765 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.396731 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.401726 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.401698 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 11 14:25:32.419500 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.419474 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h"] Sep 11 14:25:32.477154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.477114 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.477154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.477152 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwx84\" (UniqueName: \"kubernetes.io/projected/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-kube-api-access-pwx84\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.477361 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.477228 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.477361 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.477269 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.477361 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.477317 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.477462 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.477359 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.578866 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.578809 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.578866 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.578865 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwx84\" (UniqueName: \"kubernetes.io/projected/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-kube-api-access-pwx84\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.579065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.578896 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.579065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.578987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.579065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.579041 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.579208 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.579079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.579337 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.579318 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.579414 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.579373 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.579492 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.579475 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.581074 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.581053 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.581429 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.581408 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.595575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.595551 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwx84\" (UniqueName: \"kubernetes.io/projected/04f40182-4fb3-4cfc-a3aa-459d8169c0f4-kube-api-access-pwx84\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h\" (UID: \"04f40182-4fb3-4cfc-a3aa-459d8169c0f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.708067 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.707989 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:32.983433 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:32.983404 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h"] Sep 11 14:25:32.985380 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:25:32.985345 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04f40182_4fb3_4cfc_a3aa_459d8169c0f4.slice/crio-7efe4a883b417e789fbcea27d209db5616bfe0c7ea663ca8fd763a76b8d41775 WatchSource:0}: Error finding container 7efe4a883b417e789fbcea27d209db5616bfe0c7ea663ca8fd763a76b8d41775: Status 404 returned error can't find the container with id 7efe4a883b417e789fbcea27d209db5616bfe0c7ea663ca8fd763a76b8d41775 Sep 11 14:25:33.390908 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:33.390864 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" event={"ID":"04f40182-4fb3-4cfc-a3aa-459d8169c0f4","Type":"ContainerStarted","Data":"47d52af2a80c2be788e29de7dc798f79c005a0c87f57cd716eacdcf25c47954b"} Sep 11 14:25:33.391078 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:33.390917 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" event={"ID":"04f40182-4fb3-4cfc-a3aa-459d8169c0f4","Type":"ContainerStarted","Data":"7efe4a883b417e789fbcea27d209db5616bfe0c7ea663ca8fd763a76b8d41775"} Sep 11 14:25:33.414414 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:33.414364 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" podStartSLOduration=1.20310392 podStartE2EDuration="1.414350502s" totalImagesPullingTime="211.246582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:25:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:25:33.412818168 +0000 UTC m=+645.749786361" watchObservedRunningTime="2026-09-11 14:25:33.414350502 +0000 UTC m=+645.751318692" Sep 11 14:25:39.109887 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:39.109821 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:39.110292 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:39.109928 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:39.122654 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:39.122617 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:39.435157 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:39.435076 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f4sp9" Sep 11 14:25:42.708641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:42.708607 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:42.709054 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:42.708659 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:42.721202 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:42.721182 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:25:43.455864 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:25:43.455819 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h" Sep 11 14:26:30.255126 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:30.255092 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-86cb9f98d8-l4slv"] Sep 11 14:26:30.258452 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:30.258434 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86cb9f98d8-l4slv" Sep 11 14:26:30.270070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:30.270043 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86cb9f98d8-l4slv"] Sep 11 14:26:30.395635 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:30.395601 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8z9t\" (UniqueName: \"kubernetes.io/projected/302952be-9e3f-4c1c-95b6-b4d3054776cc-kube-api-access-d8z9t\") pod \"authorino-86cb9f98d8-l4slv\" (UID: \"302952be-9e3f-4c1c-95b6-b4d3054776cc\") " pod="kuadrant-system/authorino-86cb9f98d8-l4slv" Sep 11 14:26:30.395823 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:30.395648 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/302952be-9e3f-4c1c-95b6-b4d3054776cc-tls-cert\") pod \"authorino-86cb9f98d8-l4slv\" (UID: \"302952be-9e3f-4c1c-95b6-b4d3054776cc\") " pod="kuadrant-system/authorino-86cb9f98d8-l4slv" Sep 11 14:26:30.496639 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:30.496605 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8z9t\" (UniqueName: \"kubernetes.io/projected/302952be-9e3f-4c1c-95b6-b4d3054776cc-kube-api-access-d8z9t\") pod \"authorino-86cb9f98d8-l4slv\" (UID: \"302952be-9e3f-4c1c-95b6-b4d3054776cc\") " pod="kuadrant-system/authorino-86cb9f98d8-l4slv" Sep 11 14:26:30.496811 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:30.496656 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/302952be-9e3f-4c1c-95b6-b4d3054776cc-tls-cert\") pod \"authorino-86cb9f98d8-l4slv\" (UID: \"302952be-9e3f-4c1c-95b6-b4d3054776cc\") " pod="kuadrant-system/authorino-86cb9f98d8-l4slv" Sep 11 14:26:30.499099 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:30.499072 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/302952be-9e3f-4c1c-95b6-b4d3054776cc-tls-cert\") pod \"authorino-86cb9f98d8-l4slv\" (UID: \"302952be-9e3f-4c1c-95b6-b4d3054776cc\") " pod="kuadrant-system/authorino-86cb9f98d8-l4slv" Sep 11 14:26:30.506544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:30.506483 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8z9t\" (UniqueName: \"kubernetes.io/projected/302952be-9e3f-4c1c-95b6-b4d3054776cc-kube-api-access-d8z9t\") pod \"authorino-86cb9f98d8-l4slv\" (UID: \"302952be-9e3f-4c1c-95b6-b4d3054776cc\") " pod="kuadrant-system/authorino-86cb9f98d8-l4slv" Sep 11 14:26:30.567960 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:30.567936 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86cb9f98d8-l4slv" Sep 11 14:26:30.841393 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:30.841351 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86cb9f98d8-l4slv"] Sep 11 14:26:30.842296 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:26:30.842262 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod302952be_9e3f_4c1c_95b6_b4d3054776cc.slice/crio-e2b06cb7bec38661d8d155475af2b1a9f9db6284638edb943aac5b38dade8497 WatchSource:0}: Error finding container e2b06cb7bec38661d8d155475af2b1a9f9db6284638edb943aac5b38dade8497: Status 404 returned error can't find the container with id e2b06cb7bec38661d8d155475af2b1a9f9db6284638edb943aac5b38dade8497 Sep 11 14:26:31.706517 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:31.706477 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86cb9f98d8-l4slv" event={"ID":"302952be-9e3f-4c1c-95b6-b4d3054776cc","Type":"ContainerStarted","Data":"6b63caa97f506712e3337dc322993d84847ea76f21d93a1ecb4d97b64116e9b5"} Sep 11 14:26:31.707006 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:31.706530 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86cb9f98d8-l4slv" event={"ID":"302952be-9e3f-4c1c-95b6-b4d3054776cc","Type":"ContainerStarted","Data":"e2b06cb7bec38661d8d155475af2b1a9f9db6284638edb943aac5b38dade8497"} Sep 11 14:26:31.730008 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:31.729953 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-86cb9f98d8-l4slv" podStartSLOduration=1.087315713 podStartE2EDuration="1.729935778s" totalImagesPullingTime="642.620065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:26:31.728068418 +0000 UTC m=+704.065036601" watchObservedRunningTime="2026-09-11 14:26:31.729935778 +0000 UTC m=+704.066903969" Sep 11 14:26:31.785786 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:31.785705 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84cdbd95d5-mcbxc"] Sep 11 14:26:31.785970 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:31.785934 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-84cdbd95d5-mcbxc" podUID="4000e4a9-2645-49c1-a9f1-3e91dfdb333a" containerName="authorino" containerID="cri-o://cdfe8bcbf0b3e409ad237a2d4760d6e07aa96d6bf114156592fa324e7ee761ca" gracePeriod=30 Sep 11 14:26:32.053084 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:32.053009 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84cdbd95d5-mcbxc" Sep 11 14:26:32.116103 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:32.116064 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvvdk\" (UniqueName: \"kubernetes.io/projected/4000e4a9-2645-49c1-a9f1-3e91dfdb333a-kube-api-access-bvvdk\") pod \"4000e4a9-2645-49c1-a9f1-3e91dfdb333a\" (UID: \"4000e4a9-2645-49c1-a9f1-3e91dfdb333a\") " Sep 11 14:26:32.116263 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:32.116139 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4000e4a9-2645-49c1-a9f1-3e91dfdb333a-tls-cert\") pod \"4000e4a9-2645-49c1-a9f1-3e91dfdb333a\" (UID: \"4000e4a9-2645-49c1-a9f1-3e91dfdb333a\") " Sep 11 14:26:32.118218 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:32.118183 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4000e4a9-2645-49c1-a9f1-3e91dfdb333a-kube-api-access-bvvdk" (OuterVolumeSpecName: "kube-api-access-bvvdk") pod "4000e4a9-2645-49c1-a9f1-3e91dfdb333a" (UID: "4000e4a9-2645-49c1-a9f1-3e91dfdb333a"). InnerVolumeSpecName "kube-api-access-bvvdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:26:32.126722 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:32.126688 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4000e4a9-2645-49c1-a9f1-3e91dfdb333a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4000e4a9-2645-49c1-a9f1-3e91dfdb333a" (UID: "4000e4a9-2645-49c1-a9f1-3e91dfdb333a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:26:32.217377 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:32.217344 2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4000e4a9-2645-49c1-a9f1-3e91dfdb333a-tls-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:26:32.217377 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:32.217377 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvvdk\" (UniqueName: \"kubernetes.io/projected/4000e4a9-2645-49c1-a9f1-3e91dfdb333a-kube-api-access-bvvdk\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:26:32.714004 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:32.713969 2573 generic.go:356] "Generic (PLEG): container finished" podID="4000e4a9-2645-49c1-a9f1-3e91dfdb333a" containerID="cdfe8bcbf0b3e409ad237a2d4760d6e07aa96d6bf114156592fa324e7ee761ca" exitCode=0 Sep 11 14:26:32.714440 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:32.714036 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84cdbd95d5-mcbxc" Sep 11 14:26:32.714440 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:32.714054 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84cdbd95d5-mcbxc" event={"ID":"4000e4a9-2645-49c1-a9f1-3e91dfdb333a","Type":"ContainerDied","Data":"cdfe8bcbf0b3e409ad237a2d4760d6e07aa96d6bf114156592fa324e7ee761ca"} Sep 11 14:26:32.714440 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:32.714101 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84cdbd95d5-mcbxc" event={"ID":"4000e4a9-2645-49c1-a9f1-3e91dfdb333a","Type":"ContainerDied","Data":"f26a567b34a1d1707cf06958fee3e5d218485b271c8a6565884fc34d67ffd7ec"} Sep 11 14:26:32.714440 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:32.714120 2573 scope.go:117] "RemoveContainer" containerID="cdfe8bcbf0b3e409ad237a2d4760d6e07aa96d6bf114156592fa324e7ee761ca" Sep 11 14:26:32.725251 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:32.725228 2573 scope.go:117] "RemoveContainer" containerID="cdfe8bcbf0b3e409ad237a2d4760d6e07aa96d6bf114156592fa324e7ee761ca" Sep 11 14:26:32.725633 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:26:32.725605 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdfe8bcbf0b3e409ad237a2d4760d6e07aa96d6bf114156592fa324e7ee761ca\": container with ID starting with cdfe8bcbf0b3e409ad237a2d4760d6e07aa96d6bf114156592fa324e7ee761ca not found: ID does not exist" containerID="cdfe8bcbf0b3e409ad237a2d4760d6e07aa96d6bf114156592fa324e7ee761ca" Sep 11 14:26:32.725734 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:32.725642 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdfe8bcbf0b3e409ad237a2d4760d6e07aa96d6bf114156592fa324e7ee761ca"} err="failed to get container status \"cdfe8bcbf0b3e409ad237a2d4760d6e07aa96d6bf114156592fa324e7ee761ca\": rpc error: code = NotFound desc = could not find container \"cdfe8bcbf0b3e409ad237a2d4760d6e07aa96d6bf114156592fa324e7ee761ca\": container with ID starting with cdfe8bcbf0b3e409ad237a2d4760d6e07aa96d6bf114156592fa324e7ee761ca not found: ID does not exist" Sep 11 14:26:32.741723 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:32.741694 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84cdbd95d5-mcbxc"] Sep 11 14:26:32.750199 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:32.750172 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-84cdbd95d5-mcbxc"] Sep 11 14:26:34.187674 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:34.187639 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4000e4a9-2645-49c1-a9f1-3e91dfdb333a" path="/var/lib/kubelet/pods/4000e4a9-2645-49c1-a9f1-3e91dfdb333a/volumes" Sep 11 14:26:51.043658 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.043624 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd"] Sep 11 14:26:51.044059 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.044044 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4000e4a9-2645-49c1-a9f1-3e91dfdb333a" containerName="authorino" Sep 11 14:26:51.044104 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.044061 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4000e4a9-2645-49c1-a9f1-3e91dfdb333a" containerName="authorino" Sep 11 14:26:51.044146 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.044136 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4000e4a9-2645-49c1-a9f1-3e91dfdb333a" containerName="authorino" Sep 11 14:26:51.048363 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.048344 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.051490 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.051468 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a39864-openshift-default-dockercfg-rtmv5\"" Sep 11 14:26:51.061946 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.061915 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd"] Sep 11 14:26:51.085801 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.085774 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcc64\" (UniqueName: \"kubernetes.io/projected/08c123b1-afa2-4560-9bf1-bb056db36f81-kube-api-access-xcc64\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.085956 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.085804 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-envoy\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.085956 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.085823 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-workload-certs\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.085956 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.085873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-podinfo\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.085956 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.085937 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-credential-socket\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.086147 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.085978 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-workload-socket\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.086147 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.086052 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-token\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.086147 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.086069 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-data\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.086288 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.086176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c123b1-afa2-4560-9bf1-bb056db36f81-istiod-ca-cert\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.187352 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.187312 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-token\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.187352 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.187346 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-data\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.187604 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.187397 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c123b1-afa2-4560-9bf1-bb056db36f81-istiod-ca-cert\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.187604 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.187425 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcc64\" (UniqueName: \"kubernetes.io/projected/08c123b1-afa2-4560-9bf1-bb056db36f81-kube-api-access-xcc64\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.187604 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.187449 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-envoy\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.187604 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.187474 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-workload-certs\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.187604 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.187576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-podinfo\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.187943 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.187619 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-credential-socket\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.187943 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.187662 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-workload-socket\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.187943 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.187791 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-data\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.187943 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.187934 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-workload-certs\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.188150 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.188016 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-credential-socket\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.188150 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.188105 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-workload-socket\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.188254 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.188232 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c123b1-afa2-4560-9bf1-bb056db36f81-istiod-ca-cert\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.189958 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.189935 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-podinfo\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.190076 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.190058 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-envoy\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.196009 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.195982 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-token\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.196300 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.196278 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcc64\" (UniqueName: \"kubernetes.io/projected/08c123b1-afa2-4560-9bf1-bb056db36f81-kube-api-access-xcc64\") pod \"e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.359642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.359609 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:51.641787 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.641756 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd"] Sep 11 14:26:51.642885 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:26:51.642861 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c123b1_afa2_4560_9bf1_bb056db36f81.slice/crio-0caf8497dadb58ef1ed114c5127849087f0978d903f7ba0d4c0eec12ec6dd38b WatchSource:0}: Error finding container 0caf8497dadb58ef1ed114c5127849087f0978d903f7ba0d4c0eec12ec6dd38b: Status 404 returned error can't find the container with id 0caf8497dadb58ef1ed114c5127849087f0978d903f7ba0d4c0eec12ec6dd38b Sep 11 14:26:51.645356 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.645323 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:26:51.645449 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.645392 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:26:51.645449 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.645420 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:26:51.823954 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.823918 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" event={"ID":"08c123b1-afa2-4560-9bf1-bb056db36f81","Type":"ContainerStarted","Data":"1b642ba6f8f54dd4f773f9dbb2b2f32bba02335ea1d3b6434953596155d0a949"} Sep 11 14:26:51.824113 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.823968 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" event={"ID":"08c123b1-afa2-4560-9bf1-bb056db36f81","Type":"ContainerStarted","Data":"0caf8497dadb58ef1ed114c5127849087f0978d903f7ba0d4c0eec12ec6dd38b"} Sep 11 14:26:51.844186 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:51.844140 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" podStartSLOduration=0.844125645 podStartE2EDuration="844.125645ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:26:51.843059415 +0000 UTC m=+724.180027606" watchObservedRunningTime="2026-09-11 14:26:51.844125645 +0000 UTC m=+724.181093833" Sep 11 14:26:52.360029 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:52.359998 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:52.364969 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:52.364941 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:52.830269 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:52.830194 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:52.831022 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:52.831002 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:26:56.698564 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:56.698532 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7785-32c31f"] Sep 11 14:26:56.702421 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:56.702405 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7785-32c31f" Sep 11 14:26:56.705215 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:56.705179 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-998c2\"" Sep 11 14:26:56.707314 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:56.707286 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7785-32c31f"] Sep 11 14:26:56.842325 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:56.842292 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zqvf\" (UniqueName: \"kubernetes.io/projected/fd122666-80f4-48c8-bc6e-6cb92527e5e5-kube-api-access-9zqvf\") pod \"test-curl-7785-32c31f\" (UID: \"fd122666-80f4-48c8-bc6e-6cb92527e5e5\") " pod="opendatahub/test-curl-7785-32c31f" Sep 11 14:26:56.943209 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:56.943174 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zqvf\" (UniqueName: \"kubernetes.io/projected/fd122666-80f4-48c8-bc6e-6cb92527e5e5-kube-api-access-9zqvf\") pod \"test-curl-7785-32c31f\" (UID: \"fd122666-80f4-48c8-bc6e-6cb92527e5e5\") " pod="opendatahub/test-curl-7785-32c31f" Sep 11 14:26:56.953597 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:56.953533 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zqvf\" (UniqueName: \"kubernetes.io/projected/fd122666-80f4-48c8-bc6e-6cb92527e5e5-kube-api-access-9zqvf\") pod \"test-curl-7785-32c31f\" (UID: \"fd122666-80f4-48c8-bc6e-6cb92527e5e5\") " pod="opendatahub/test-curl-7785-32c31f" Sep 11 14:26:57.013566 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:57.013534 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7785-32c31f" Sep 11 14:26:57.305914 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:57.305822 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7785-32c31f"] Sep 11 14:26:57.309222 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:26:57.309198 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd122666_80f4_48c8_bc6e_6cb92527e5e5.slice/crio-e30a83e7243bc042d2ecb31d2f393133dfd1de3aa633fdc5d3137c7c51d8c1a4 WatchSource:0}: Error finding container e30a83e7243bc042d2ecb31d2f393133dfd1de3aa633fdc5d3137c7c51d8c1a4: Status 404 returned error can't find the container with id e30a83e7243bc042d2ecb31d2f393133dfd1de3aa633fdc5d3137c7c51d8c1a4 Sep 11 14:26:57.860549 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:57.860514 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7785-32c31f" event={"ID":"fd122666-80f4-48c8-bc6e-6cb92527e5e5","Type":"ContainerStarted","Data":"19a10ffc90372b5c09a69c97573e4a27ec7d88766d0b5548a497fa3e434e982e"} Sep 11 14:26:57.861052 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:57.860565 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7785-32c31f" event={"ID":"fd122666-80f4-48c8-bc6e-6cb92527e5e5","Type":"ContainerStarted","Data":"e30a83e7243bc042d2ecb31d2f393133dfd1de3aa633fdc5d3137c7c51d8c1a4"} Sep 11 14:26:57.880537 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:57.880493 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7785-32c31f" podStartSLOduration=1.421192544 podStartE2EDuration="1.880479371s" totalImagesPullingTime="459.286827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:26:57.878697971 +0000 UTC m=+730.215666162" watchObservedRunningTime="2026-09-11 14:26:57.880479371 +0000 UTC m=+730.217447562" Sep 11 14:26:58.522538 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:58.522503 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7785-32c31f"] Sep 11 14:26:58.527434 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:58.527405 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7785-32c31f"] Sep 11 14:26:58.866185 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:26:58.866155 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7785-32c31f" container="test-curl-7785-32c31f" Sep 11 14:26:59.417167 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.417135 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz"] Sep 11 14:26:59.417680 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.417663 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd122666-80f4-48c8-bc6e-6cb92527e5e5" containerName="test-curl-7785-32c31f" Sep 11 14:26:59.417736 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.417682 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd122666-80f4-48c8-bc6e-6cb92527e5e5" containerName="test-curl-7785-32c31f" Sep 11 14:26:59.417783 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.417756 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd122666-80f4-48c8-bc6e-6cb92527e5e5" containerName="test-curl-7785-32c31f" Sep 11 14:26:59.421086 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.421068 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.424093 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.424068 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-b4a233-openshift-default-dockercfg-l84pq\"" Sep 11 14:26:59.439571 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.439546 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz"] Sep 11 14:26:59.565553 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.565512 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-data\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.565759 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.565567 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istiod-ca-cert\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.565759 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.565692 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-token\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.565759 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.565731 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-workload-certs\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.565993 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.565760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-credential-socket\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.565993 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.565794 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-podinfo\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.565993 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.565855 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-workload-socket\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.565993 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.565919 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxhq8\" (UniqueName: \"kubernetes.io/projected/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-kube-api-access-sxhq8\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.565993 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.565981 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-envoy\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.667383 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.667291 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-envoy\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.667383 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.667361 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-data\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.667635 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.667406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istiod-ca-cert\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.667635 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.667479 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-token\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.667635 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.667511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-workload-certs\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.667635 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.667535 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-credential-socket\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.667635 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.667567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-podinfo\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.667635 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.667617 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-workload-socket\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.667980 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.667692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxhq8\" (UniqueName: \"kubernetes.io/projected/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-kube-api-access-sxhq8\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.668055 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.668024 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-workload-certs\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.668119 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.668095 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-credential-socket\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.668230 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.668183 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istiod-ca-cert\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.668343 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.668320 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-data\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.668620 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.668600 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-workload-socket\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.670749 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.670723 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-envoy\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.671174 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.671150 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-podinfo\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.677975 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.677946 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-token\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.678245 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.678223 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxhq8\" (UniqueName: \"kubernetes.io/projected/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-kube-api-access-sxhq8\") pod \"e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.733089 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.733055 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:26:59.853845 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.853789 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v"] Sep 11 14:26:59.858600 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.858576 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:26:59.866775 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.866749 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-43ca2273-openshift-default-dockercfg-z6ls7\"" Sep 11 14:26:59.873275 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.873246 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7785-32c31f" podUID="fd122666-80f4-48c8-bc6e-6cb92527e5e5" containerName="test-curl-7785-32c31f" containerID="cri-o://19a10ffc90372b5c09a69c97573e4a27ec7d88766d0b5548a497fa3e434e982e" gracePeriod=2 Sep 11 14:26:59.893147 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.893114 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v"] Sep 11 14:26:59.923619 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:26:59.923545 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7785-32c31f\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" podUID="fd122666-80f4-48c8-bc6e-6cb92527e5e5" pod="opendatahub/test-curl-7785-32c31f" Sep 11 14:26:59.971204 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.971165 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-credential-socket\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:26:59.971355 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.971220 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02685af-012f-403f-a6cc-e5ad699bb2e2-istiod-ca-cert\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:26:59.971355 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.971287 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tvcl\" (UniqueName: \"kubernetes.io/projected/c02685af-012f-403f-a6cc-e5ad699bb2e2-kube-api-access-5tvcl\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:26:59.971453 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.971399 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-podinfo\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:26:59.971487 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.971462 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-token\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:26:59.971533 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.971508 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-data\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:26:59.971587 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.971537 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-workload-certs\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:26:59.971587 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.971581 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-envoy\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:26:59.971668 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:26:59.971615 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-workload-socket\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.026293 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.026262 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz"] Sep 11 14:27:00.027214 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:27:00.027179 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4d46c5_9e56_4a4c_86d4_d5caeb1a412d.slice/crio-426c5b5fce7221f278e9374e66951e916edc49c2c9e0cc9de2ac112216cbce28 WatchSource:0}: Error finding container 426c5b5fce7221f278e9374e66951e916edc49c2c9e0cc9de2ac112216cbce28: Status 404 returned error can't find the container with id 426c5b5fce7221f278e9374e66951e916edc49c2c9e0cc9de2ac112216cbce28 Sep 11 14:27:00.029611 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.029577 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:00.029703 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.029644 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:00.029703 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.029670 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:00.072186 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.072163 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-workload-socket\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.072315 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.072228 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-credential-socket\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.072315 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.072249 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02685af-012f-403f-a6cc-e5ad699bb2e2-istiod-ca-cert\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.072315 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.072285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tvcl\" (UniqueName: \"kubernetes.io/projected/c02685af-012f-403f-a6cc-e5ad699bb2e2-kube-api-access-5tvcl\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.072488 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.072352 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-podinfo\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.072488 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.072393 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-token\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.072488 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.072436 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-data\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.072488 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.072466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-workload-certs\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.072715 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.072519 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-envoy\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.072715 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.072622 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-workload-socket\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.072715 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.072652 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-credential-socket\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.073010 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.072989 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-data\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.073160 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.073085 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-workload-certs\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.073532 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.073509 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02685af-012f-403f-a6cc-e5ad699bb2e2-istiod-ca-cert\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.074764 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.074740 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-envoy\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.075083 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.075065 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-podinfo\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.081483 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.081452 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tvcl\" (UniqueName: \"kubernetes.io/projected/c02685af-012f-403f-a6cc-e5ad699bb2e2-kube-api-access-5tvcl\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.081552 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.081535 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-token\") pod \"e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.170484 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.170452 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:00.459167 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.459133 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v"] Sep 11 14:27:00.459906 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:27:00.459876 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02685af_012f_403f_a6cc_e5ad699bb2e2.slice/crio-366ac64b414583939bdb0d1472038eba80bf2cfc96dcba8e1504848087462021 WatchSource:0}: Error finding container 366ac64b414583939bdb0d1472038eba80bf2cfc96dcba8e1504848087462021: Status 404 returned error can't find the container with id 366ac64b414583939bdb0d1472038eba80bf2cfc96dcba8e1504848087462021 Sep 11 14:27:00.462387 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.462351 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:00.462485 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.462428 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:00.462485 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.462472 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:00.883048 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.883002 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" event={"ID":"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d","Type":"ContainerStarted","Data":"cc2da33b2ef3736e9361040c67b44b61d49257f0ef3160967c5e2b0bf8435f5e"} Sep 11 14:27:00.883495 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.883076 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" event={"ID":"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d","Type":"ContainerStarted","Data":"426c5b5fce7221f278e9374e66951e916edc49c2c9e0cc9de2ac112216cbce28"} Sep 11 14:27:00.886238 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.886190 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" event={"ID":"c02685af-012f-403f-a6cc-e5ad699bb2e2","Type":"ContainerStarted","Data":"8663c8f5286410c71c8a17c542196f6e519e65103b4de9f8266c654e50d8e8ea"} Sep 11 14:27:00.886374 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.886267 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" event={"ID":"c02685af-012f-403f-a6cc-e5ad699bb2e2","Type":"ContainerStarted","Data":"366ac64b414583939bdb0d1472038eba80bf2cfc96dcba8e1504848087462021"} Sep 11 14:27:00.909312 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.909247 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" podStartSLOduration=1.909225394 podStartE2EDuration="1.909225394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:27:00.906844868 +0000 UTC m=+733.243813052" watchObservedRunningTime="2026-09-11 14:27:00.909225394 +0000 UTC m=+733.246193588" Sep 11 14:27:00.936267 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:00.936193 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" podStartSLOduration=1.936171806 podStartE2EDuration="1.936171806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:27:00.932340339 +0000 UTC m=+733.269308542" watchObservedRunningTime="2026-09-11 14:27:00.936171806 +0000 UTC m=+733.273139998" Sep 11 14:27:01.170879 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:01.170770 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:01.172185 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:01.172161 2573 patch_prober.go:28] interesting pod/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 14:27:01.172277 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:01.172210 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" podUID="c02685af-012f-403f-a6cc-e5ad699bb2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 14:27:01.734088 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:01.733996 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:27:01.747952 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:01.747919 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:27:01.892111 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:01.892067 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:27:01.892859 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:01.892819 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:27:02.106937 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:02.106911 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7785-32c31f" Sep 11 14:27:02.175232 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:02.175203 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:02.198387 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:02.198362 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zqvf\" (UniqueName: \"kubernetes.io/projected/fd122666-80f4-48c8-bc6e-6cb92527e5e5-kube-api-access-9zqvf\") pod \"fd122666-80f4-48c8-bc6e-6cb92527e5e5\" (UID: \"fd122666-80f4-48c8-bc6e-6cb92527e5e5\") " Sep 11 14:27:02.200455 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:02.200434 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd122666-80f4-48c8-bc6e-6cb92527e5e5-kube-api-access-9zqvf" (OuterVolumeSpecName: "kube-api-access-9zqvf") pod "fd122666-80f4-48c8-bc6e-6cb92527e5e5" (UID: "fd122666-80f4-48c8-bc6e-6cb92527e5e5"). InnerVolumeSpecName "kube-api-access-9zqvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:27:02.299880 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:02.299782 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zqvf\" (UniqueName: \"kubernetes.io/projected/fd122666-80f4-48c8-bc6e-6cb92527e5e5-kube-api-access-9zqvf\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:27:02.899532 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:02.899483 2573 generic.go:356] "Generic (PLEG): container finished" podID="fd122666-80f4-48c8-bc6e-6cb92527e5e5" containerID="19a10ffc90372b5c09a69c97573e4a27ec7d88766d0b5548a497fa3e434e982e" exitCode=137 Sep 11 14:27:02.899532 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:02.899526 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7785-32c31f" Sep 11 14:27:02.900169 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:02.899588 2573 scope.go:117] "RemoveContainer" containerID="19a10ffc90372b5c09a69c97573e4a27ec7d88766d0b5548a497fa3e434e982e" Sep 11 14:27:02.900169 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:02.900100 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:02.901774 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:02.901751 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:02.912227 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:02.912070 2573 scope.go:117] "RemoveContainer" containerID="19a10ffc90372b5c09a69c97573e4a27ec7d88766d0b5548a497fa3e434e982e" Sep 11 14:27:02.912520 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:27:02.912502 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a10ffc90372b5c09a69c97573e4a27ec7d88766d0b5548a497fa3e434e982e\": container with ID starting with 19a10ffc90372b5c09a69c97573e4a27ec7d88766d0b5548a497fa3e434e982e not found: ID does not exist" containerID="19a10ffc90372b5c09a69c97573e4a27ec7d88766d0b5548a497fa3e434e982e" Sep 11 14:27:02.912572 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:02.912526 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a10ffc90372b5c09a69c97573e4a27ec7d88766d0b5548a497fa3e434e982e"} err="failed to get container status \"19a10ffc90372b5c09a69c97573e4a27ec7d88766d0b5548a497fa3e434e982e\": rpc error: code = NotFound desc = could not find container \"19a10ffc90372b5c09a69c97573e4a27ec7d88766d0b5548a497fa3e434e982e\": container with ID starting with 19a10ffc90372b5c09a69c97573e4a27ec7d88766d0b5548a497fa3e434e982e not found: ID does not exist" Sep 11 14:27:04.187736 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:04.187706 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd122666-80f4-48c8-bc6e-6cb92527e5e5" path="/var/lib/kubelet/pods/fd122666-80f4-48c8-bc6e-6cb92527e5e5/volumes" Sep 11 14:27:08.871999 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:08.871941 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 14:27:08.872493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:08.872023 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" podUID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 14:27:09.441081 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.441010 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm"] Sep 11 14:27:09.444589 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.444571 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.447249 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.447228 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b6b864-openshift-default-dockercfg-45pkl\"" Sep 11 14:27:09.458604 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.458582 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm"] Sep 11 14:27:09.581171 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.581135 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-istio-data\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.581171 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.581175 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbzpp\" (UniqueName: \"kubernetes.io/projected/3d2a384c-8016-4611-b381-df0695e86d41-kube-api-access-hbzpp\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.581385 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.581198 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-workload-certs\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.581385 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.581266 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2a384c-8016-4611-b381-df0695e86d41-istiod-ca-cert\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.581385 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.581339 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-workload-socket\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.581385 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.581371 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2a384c-8016-4611-b381-df0695e86d41-istio-podinfo\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.581520 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.581434 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-istio-envoy\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.581520 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.581493 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-credential-socket\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.581584 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.581537 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2a384c-8016-4611-b381-df0695e86d41-istio-token\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.682312 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.682262 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-istio-envoy\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.682493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.682359 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-credential-socket\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.682493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.682419 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2a384c-8016-4611-b381-df0695e86d41-istio-token\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.682626 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.682498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-istio-data\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.682703 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.682666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzpp\" (UniqueName: \"kubernetes.io/projected/3d2a384c-8016-4611-b381-df0695e86d41-kube-api-access-hbzpp\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.682770 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.682722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-workload-certs\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.682770 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.682762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2a384c-8016-4611-b381-df0695e86d41-istiod-ca-cert\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.682906 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.682793 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-workload-socket\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.682906 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.682819 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2a384c-8016-4611-b381-df0695e86d41-istio-podinfo\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.682906 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.682855 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-credential-socket\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.683059 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.682957 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-istio-data\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.683119 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.683101 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-workload-certs\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.683406 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.683367 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-workload-socket\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.683717 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.683696 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2a384c-8016-4611-b381-df0695e86d41-istiod-ca-cert\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.684891 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.684866 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-istio-envoy\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.685307 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.685290 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2a384c-8016-4611-b381-df0695e86d41-istio-podinfo\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.696239 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.696171 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbzpp\" (UniqueName: \"kubernetes.io/projected/3d2a384c-8016-4611-b381-df0695e86d41-kube-api-access-hbzpp\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.697505 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.697480 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2a384c-8016-4611-b381-df0695e86d41-istio-token\") pod \"e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.754912 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.754824 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:09.942562 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.942530 2573 generic.go:356] "Generic (PLEG): container finished" podID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerID="1b642ba6f8f54dd4f773f9dbb2b2f32bba02335ea1d3b6434953596155d0a949" exitCode=0 Sep 11 14:27:09.942938 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.942602 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" event={"ID":"08c123b1-afa2-4560-9bf1-bb056db36f81","Type":"ContainerDied","Data":"1b642ba6f8f54dd4f773f9dbb2b2f32bba02335ea1d3b6434953596155d0a949"} Sep 11 14:27:09.943083 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.943068 2573 scope.go:117] "RemoveContainer" containerID="1b642ba6f8f54dd4f773f9dbb2b2f32bba02335ea1d3b6434953596155d0a949" Sep 11 14:27:09.944305 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.944276 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:09.944394 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.944343 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:09.944394 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:09.944374 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:10.101530 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:27:10.101488 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2a384c_8016_4611_b381_df0695e86d41.slice/crio-2b772f99491d602b4f437ca675963e15479ea41cfb92badb00a5b16313465646 WatchSource:0}: Error finding container 2b772f99491d602b4f437ca675963e15479ea41cfb92badb00a5b16313465646: Status 404 returned error can't find the container with id 2b772f99491d602b4f437ca675963e15479ea41cfb92badb00a5b16313465646 Sep 11 14:27:10.101761 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:10.101744 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm"] Sep 11 14:27:10.104714 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:10.104668 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:10.104885 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:10.104794 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:10.104885 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:10.104876 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:10.951506 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:10.951476 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" event={"ID":"08c123b1-afa2-4560-9bf1-bb056db36f81","Type":"ContainerStarted","Data":"e5568f1fe185f6c35721d760934d2296a933007c57574c71eab154d17a72103a"} Sep 11 14:27:10.953202 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:10.953176 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" event={"ID":"3d2a384c-8016-4611-b381-df0695e86d41","Type":"ContainerStarted","Data":"4d6103063d5115921f1e5c27aeb8396ed5f45b4fc5094c73e7e84a589081e688"} Sep 11 14:27:10.953366 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:10.953222 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" event={"ID":"3d2a384c-8016-4611-b381-df0695e86d41","Type":"ContainerStarted","Data":"2b772f99491d602b4f437ca675963e15479ea41cfb92badb00a5b16313465646"} Sep 11 14:27:11.008178 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:11.008114 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" podStartSLOduration=2.008094062 podStartE2EDuration="2.008094062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:27:11.004471028 +0000 UTC m=+743.341439217" watchObservedRunningTime="2026-09-11 14:27:11.008094062 +0000 UTC m=+743.345062259" Sep 11 14:27:11.360820 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:11.360777 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:27:11.361158 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:11.361125 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 14:27:11.361387 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:11.361357 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" podUID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 14:27:11.758044 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:11.757939 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:11.765886 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:11.765827 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:11.963930 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:11.963902 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:11.967938 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:11.967915 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:12.360643 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:12.360604 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 14:27:12.360989 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:12.360943 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" podUID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 14:27:13.360388 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:13.360348 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 14:27:13.361000 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:13.360961 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" podUID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 14:27:14.360804 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:14.360766 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 14:27:14.361791 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:14.361380 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" podUID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 14:27:15.361102 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:15.360641 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 14:27:15.361102 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:15.360704 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" podUID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 14:27:16.361668 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:16.360522 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 14:27:16.361668 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:16.360588 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" podUID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 14:27:17.363609 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:17.363577 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:27:17.364072 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:17.364058 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:27:17.364810 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:17.364790 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:27:17.935010 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:17.934959 2573 patch_prober.go:28] interesting pod/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 14:27:17.935151 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:17.935030 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" podUID="2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 14:27:21.045439 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:21.045399 2573 generic.go:356] "Generic (PLEG): container finished" podID="c02685af-012f-403f-a6cc-e5ad699bb2e2" containerID="8663c8f5286410c71c8a17c542196f6e519e65103b4de9f8266c654e50d8e8ea" exitCode=0 Sep 11 14:27:21.045901 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:21.045468 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" event={"ID":"c02685af-012f-403f-a6cc-e5ad699bb2e2","Type":"ContainerDied","Data":"8663c8f5286410c71c8a17c542196f6e519e65103b4de9f8266c654e50d8e8ea"} Sep 11 14:27:21.046092 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:21.046071 2573 scope.go:117] "RemoveContainer" containerID="8663c8f5286410c71c8a17c542196f6e519e65103b4de9f8266c654e50d8e8ea" Sep 11 14:27:21.047405 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:21.047373 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:21.047405 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:21.047399 2573 generic.go:356] "Generic (PLEG): container finished" podID="2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" containerID="cc2da33b2ef3736e9361040c67b44b61d49257f0ef3160967c5e2b0bf8435f5e" exitCode=0 Sep 11 14:27:21.047682 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:21.047452 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:21.047682 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:21.047470 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" event={"ID":"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d","Type":"ContainerDied","Data":"cc2da33b2ef3736e9361040c67b44b61d49257f0ef3160967c5e2b0bf8435f5e"} Sep 11 14:27:21.047682 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:21.047482 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:21.047929 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:21.047913 2573 scope.go:117] "RemoveContainer" containerID="cc2da33b2ef3736e9361040c67b44b61d49257f0ef3160967c5e2b0bf8435f5e" Sep 11 14:27:21.049008 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:21.048985 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:21.049082 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:21.049043 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:21.049082 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:21.049073 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:22.060955 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:22.060915 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" event={"ID":"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d","Type":"ContainerStarted","Data":"d5b7dd6b701b6080ce3549c7201e96c1911cd411c0049e0da3b1de7776c47a35"} Sep 11 14:27:22.068950 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:22.068893 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" event={"ID":"c02685af-012f-403f-a6cc-e5ad699bb2e2","Type":"ContainerStarted","Data":"77529c3ac690066868457c21ed89da38891bcd7419cb7bb43a15413ac9a1013e"} Sep 11 14:27:22.171318 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:22.171276 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:22.171529 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:22.171429 2573 patch_prober.go:28] interesting pod/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 14:27:22.171529 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:22.171479 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" podUID="c02685af-012f-403f-a6cc-e5ad699bb2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 14:27:22.733474 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:22.733429 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:27:22.733794 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:22.733485 2573 patch_prober.go:28] interesting pod/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:27:22.734034 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:22.733980 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" podUID="2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 14:27:23.173256 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:23.173217 2573 patch_prober.go:28] interesting pod/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 14:27:23.173849 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:23.173797 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" podUID="c02685af-012f-403f-a6cc-e5ad699bb2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 14:27:23.733683 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:23.733642 2573 patch_prober.go:28] interesting pod/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:27:23.734028 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:23.733988 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" podUID="2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 14:27:24.171590 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:24.171553 2573 patch_prober.go:28] interesting pod/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 14:27:24.171906 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:24.171873 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" podUID="c02685af-012f-403f-a6cc-e5ad699bb2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 14:27:24.734463 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:24.734415 2573 patch_prober.go:28] interesting pod/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:27:24.734948 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:24.734487 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" podUID="2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 14:27:25.171856 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:25.171200 2573 patch_prober.go:28] interesting pod/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 14:27:25.171856 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:25.171264 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" podUID="c02685af-012f-403f-a6cc-e5ad699bb2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 14:27:25.734058 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:25.734021 2573 patch_prober.go:28] interesting pod/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:27:25.734349 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:25.734322 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" podUID="2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 14:27:26.171649 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:26.171603 2573 patch_prober.go:28] interesting pod/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 14:27:26.172169 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:26.171675 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" podUID="c02685af-012f-403f-a6cc-e5ad699bb2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 14:27:26.734010 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:26.733972 2573 patch_prober.go:28] interesting pod/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:27:26.734323 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:26.734290 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" podUID="2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 14:27:27.172059 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:27.171564 2573 patch_prober.go:28] interesting pod/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 14:27:27.172059 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:27.171635 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" podUID="c02685af-012f-403f-a6cc-e5ad699bb2e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 14:27:27.743213 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:27.743169 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:27:27.744347 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:27.744291 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:27:27.747365 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:27.747347 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:27:28.176024 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:28.175989 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:28.176493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:28.176308 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:28.177279 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:28.177256 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:28.714814 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:28.714778 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v"] Sep 11 14:27:29.119933 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.119900 2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" secret="" err="secret \"e2e-ait-43ca2273-openshift-default-dockercfg-z6ls7\" not found" Sep 11 14:27:29.745944 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.745889 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p"] Sep 11 14:27:29.754317 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.754280 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.757406 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.757382 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-2db3d4c3-gw-openshift-default-dockercfg-s7hhq\"" Sep 11 14:27:29.764044 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.764015 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p"] Sep 11 14:27:29.815562 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.815520 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh42p\" (UniqueName: \"kubernetes.io/projected/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-kube-api-access-gh42p\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.815735 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.815575 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-workload-certs\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.815735 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.815603 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-envoy\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.815735 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.815637 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istiod-ca-cert\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.815735 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.815720 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-podinfo\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.815928 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.815746 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-token\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.815928 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.815784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-workload-socket\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.815928 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.815811 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-credential-socket\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.815928 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.815856 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-data\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.917029 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.916996 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh42p\" (UniqueName: \"kubernetes.io/projected/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-kube-api-access-gh42p\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.917029 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.917035 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-workload-certs\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.917272 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.917060 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-envoy\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.917272 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.917084 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istiod-ca-cert\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.917272 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.917134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-podinfo\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.917272 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.917158 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-token\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.917515 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.917263 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-workload-socket\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.917515 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.917342 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-credential-socket\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.917515 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.917371 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-data\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.917673 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.917630 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-workload-certs\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.917732 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.917694 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-workload-socket\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.917808 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.917786 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-credential-socket\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.917928 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.917900 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-data\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.918135 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.918101 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istiod-ca-cert\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.919549 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.919526 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-envoy\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.919700 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.919684 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-podinfo\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.929362 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.929314 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-token\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:29.929486 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:29.929472 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh42p\" (UniqueName: \"kubernetes.io/projected/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-kube-api-access-gh42p\") pod \"e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:30.067156 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:30.067072 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:30.139876 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:30.138937 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" podUID="c02685af-012f-403f-a6cc-e5ad699bb2e2" containerName="istio-proxy" containerID="cri-o://77529c3ac690066868457c21ed89da38891bcd7419cb7bb43a15413ac9a1013e" gracePeriod=30 Sep 11 14:27:30.364293 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:30.364263 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p"] Sep 11 14:27:30.365267 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:27:30.365242 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18fa0554_6c12_40a1_b97c_e82f9a03a6f3.slice/crio-15bea6dc4645ee44e850aad24079e0741350866ac39dd992cfaa93035470bcbe WatchSource:0}: Error finding container 15bea6dc4645ee44e850aad24079e0741350866ac39dd992cfaa93035470bcbe: Status 404 returned error can't find the container with id 15bea6dc4645ee44e850aad24079e0741350866ac39dd992cfaa93035470bcbe Sep 11 14:27:30.367968 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:30.367937 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:30.368063 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:30.368001 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:30.368063 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:30.368032 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:31.151443 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:31.151406 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" event={"ID":"18fa0554-6c12-40a1-b97c-e82f9a03a6f3","Type":"ContainerStarted","Data":"c5d3be901f6b6b8b2c8d327a94cc43592cc170b4097e894131950c3221a59e5d"} Sep 11 14:27:31.151941 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:31.151489 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" event={"ID":"18fa0554-6c12-40a1-b97c-e82f9a03a6f3","Type":"ContainerStarted","Data":"15bea6dc4645ee44e850aad24079e0741350866ac39dd992cfaa93035470bcbe"} Sep 11 14:27:31.182996 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:31.182935 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" podStartSLOduration=2.182916116 podStartE2EDuration="2.182916116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:27:31.182595823 +0000 UTC m=+763.519564015" watchObservedRunningTime="2026-09-11 14:27:31.182916116 +0000 UTC m=+763.519884308" Sep 11 14:27:32.067456 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:32.067422 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:32.072811 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:32.072784 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:32.158905 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:32.158869 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:32.160053 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:32.160030 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:35.817061 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.817037 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:35.889387 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.889355 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-workload-socket\") pod \"c02685af-012f-403f-a6cc-e5ad699bb2e2\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " Sep 11 14:27:35.889561 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.889421 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tvcl\" (UniqueName: \"kubernetes.io/projected/c02685af-012f-403f-a6cc-e5ad699bb2e2-kube-api-access-5tvcl\") pod \"c02685af-012f-403f-a6cc-e5ad699bb2e2\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " Sep 11 14:27:35.889561 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.889447 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-token\") pod \"c02685af-012f-403f-a6cc-e5ad699bb2e2\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " Sep 11 14:27:35.889561 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.889485 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-workload-certs\") pod \"c02685af-012f-403f-a6cc-e5ad699bb2e2\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " Sep 11 14:27:35.889561 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.889508 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-envoy\") pod \"c02685af-012f-403f-a6cc-e5ad699bb2e2\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " Sep 11 14:27:35.889783 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.889606 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-data\") pod \"c02685af-012f-403f-a6cc-e5ad699bb2e2\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " Sep 11 14:27:35.889783 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.889634 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c02685af-012f-403f-a6cc-e5ad699bb2e2" (UID: "c02685af-012f-403f-a6cc-e5ad699bb2e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:27:35.889783 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.889649 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02685af-012f-403f-a6cc-e5ad699bb2e2-istiod-ca-cert\") pod \"c02685af-012f-403f-a6cc-e5ad699bb2e2\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " Sep 11 14:27:35.889783 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.889737 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-credential-socket\") pod \"c02685af-012f-403f-a6cc-e5ad699bb2e2\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " Sep 11 14:27:35.890122 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.889775 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-podinfo\") pod \"c02685af-012f-403f-a6cc-e5ad699bb2e2\" (UID: \"c02685af-012f-403f-a6cc-e5ad699bb2e2\") " Sep 11 14:27:35.890122 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.889889 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-data" (OuterVolumeSpecName: "istio-data") pod "c02685af-012f-403f-a6cc-e5ad699bb2e2" (UID: "c02685af-012f-403f-a6cc-e5ad699bb2e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:27:35.890122 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.889910 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c02685af-012f-403f-a6cc-e5ad699bb2e2" (UID: "c02685af-012f-403f-a6cc-e5ad699bb2e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:27:35.890122 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.889990 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c02685af-012f-403f-a6cc-e5ad699bb2e2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c02685af-012f-403f-a6cc-e5ad699bb2e2" (UID: "c02685af-012f-403f-a6cc-e5ad699bb2e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:27:35.890122 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.890101 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c02685af-012f-403f-a6cc-e5ad699bb2e2" (UID: "c02685af-012f-403f-a6cc-e5ad699bb2e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:27:35.890404 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.890379 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:27:35.890486 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.890408 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:27:35.890486 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.890422 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:27:35.890486 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.890436 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02685af-012f-403f-a6cc-e5ad699bb2e2-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:27:35.890486 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.890449 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:27:35.891730 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.891700 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c02685af-012f-403f-a6cc-e5ad699bb2e2" (UID: "c02685af-012f-403f-a6cc-e5ad699bb2e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:27:35.891730 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.891716 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-token" (OuterVolumeSpecName: "istio-token") pod "c02685af-012f-403f-a6cc-e5ad699bb2e2" (UID: "c02685af-012f-403f-a6cc-e5ad699bb2e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:27:35.891876 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.891782 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02685af-012f-403f-a6cc-e5ad699bb2e2-kube-api-access-5tvcl" (OuterVolumeSpecName: "kube-api-access-5tvcl") pod "c02685af-012f-403f-a6cc-e5ad699bb2e2" (UID: "c02685af-012f-403f-a6cc-e5ad699bb2e2"). InnerVolumeSpecName "kube-api-access-5tvcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:27:35.891996 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.891978 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c02685af-012f-403f-a6cc-e5ad699bb2e2" (UID: "c02685af-012f-403f-a6cc-e5ad699bb2e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:27:35.991963 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.991919 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:27:35.991963 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.991959 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tvcl\" (UniqueName: \"kubernetes.io/projected/c02685af-012f-403f-a6cc-e5ad699bb2e2-kube-api-access-5tvcl\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:27:35.991963 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.991969 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:27:35.991963 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:35.991977 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02685af-012f-403f-a6cc-e5ad699bb2e2-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:27:36.186010 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.185968 2573 generic.go:356] "Generic (PLEG): container finished" podID="3d2a384c-8016-4611-b381-df0695e86d41" containerID="4d6103063d5115921f1e5c27aeb8396ed5f45b4fc5094c73e7e84a589081e688" exitCode=0 Sep 11 14:27:36.188086 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.188060 2573 generic.go:356] "Generic (PLEG): container finished" podID="c02685af-012f-403f-a6cc-e5ad699bb2e2" containerID="77529c3ac690066868457c21ed89da38891bcd7419cb7bb43a15413ac9a1013e" exitCode=0 Sep 11 14:27:36.188205 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.188133 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" event={"ID":"3d2a384c-8016-4611-b381-df0695e86d41","Type":"ContainerDied","Data":"4d6103063d5115921f1e5c27aeb8396ed5f45b4fc5094c73e7e84a589081e688"} Sep 11 14:27:36.188205 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.188178 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" event={"ID":"c02685af-012f-403f-a6cc-e5ad699bb2e2","Type":"ContainerDied","Data":"77529c3ac690066868457c21ed89da38891bcd7419cb7bb43a15413ac9a1013e"} Sep 11 14:27:36.188205 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.188145 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" Sep 11 14:27:36.188205 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.188199 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v" event={"ID":"c02685af-012f-403f-a6cc-e5ad699bb2e2","Type":"ContainerDied","Data":"366ac64b414583939bdb0d1472038eba80bf2cfc96dcba8e1504848087462021"} Sep 11 14:27:36.188431 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.188259 2573 scope.go:117] "RemoveContainer" containerID="77529c3ac690066868457c21ed89da38891bcd7419cb7bb43a15413ac9a1013e" Sep 11 14:27:36.189042 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.189025 2573 scope.go:117] "RemoveContainer" containerID="4d6103063d5115921f1e5c27aeb8396ed5f45b4fc5094c73e7e84a589081e688" Sep 11 14:27:36.190311 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.190278 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:36.190441 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.190361 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:36.190441 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.190407 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:36.204092 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.204068 2573 scope.go:117] "RemoveContainer" containerID="8663c8f5286410c71c8a17c542196f6e519e65103b4de9f8266c654e50d8e8ea" Sep 11 14:27:36.221963 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.221943 2573 scope.go:117] "RemoveContainer" containerID="77529c3ac690066868457c21ed89da38891bcd7419cb7bb43a15413ac9a1013e" Sep 11 14:27:36.222255 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:27:36.222235 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77529c3ac690066868457c21ed89da38891bcd7419cb7bb43a15413ac9a1013e\": container with ID starting with 77529c3ac690066868457c21ed89da38891bcd7419cb7bb43a15413ac9a1013e not found: ID does not exist" containerID="77529c3ac690066868457c21ed89da38891bcd7419cb7bb43a15413ac9a1013e" Sep 11 14:27:36.222344 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.222267 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77529c3ac690066868457c21ed89da38891bcd7419cb7bb43a15413ac9a1013e"} err="failed to get container status \"77529c3ac690066868457c21ed89da38891bcd7419cb7bb43a15413ac9a1013e\": rpc error: code = NotFound desc = could not find container \"77529c3ac690066868457c21ed89da38891bcd7419cb7bb43a15413ac9a1013e\": container with ID starting with 77529c3ac690066868457c21ed89da38891bcd7419cb7bb43a15413ac9a1013e not found: ID does not exist" Sep 11 14:27:36.222344 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.222295 2573 scope.go:117] "RemoveContainer" containerID="8663c8f5286410c71c8a17c542196f6e519e65103b4de9f8266c654e50d8e8ea" Sep 11 14:27:36.222569 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:27:36.222541 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8663c8f5286410c71c8a17c542196f6e519e65103b4de9f8266c654e50d8e8ea\": container with ID starting with 8663c8f5286410c71c8a17c542196f6e519e65103b4de9f8266c654e50d8e8ea not found: ID does not exist" containerID="8663c8f5286410c71c8a17c542196f6e519e65103b4de9f8266c654e50d8e8ea" Sep 11 14:27:36.222612 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.222579 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8663c8f5286410c71c8a17c542196f6e519e65103b4de9f8266c654e50d8e8ea"} err="failed to get container status \"8663c8f5286410c71c8a17c542196f6e519e65103b4de9f8266c654e50d8e8ea\": rpc error: code = NotFound desc = could not find container \"8663c8f5286410c71c8a17c542196f6e519e65103b4de9f8266c654e50d8e8ea\": container with ID starting with 8663c8f5286410c71c8a17c542196f6e519e65103b4de9f8266c654e50d8e8ea not found: ID does not exist" Sep 11 14:27:36.240974 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.240951 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v"] Sep 11 14:27:36.245018 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:36.244992 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-43ca2273-openshift-default-6656f9d49d-k8b2v"] Sep 11 14:27:37.196817 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:37.196786 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" event={"ID":"3d2a384c-8016-4611-b381-df0695e86d41","Type":"ContainerStarted","Data":"698bd5e12e2711963de6fe084b7dc26a6b2fd281c7e0826ff84506d7e9aee4db"} Sep 11 14:27:37.755752 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:37.755719 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:37.756758 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:37.756733 2573 patch_prober.go:28] interesting pod/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 14:27:37.756888 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:37.756793 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" podUID="3d2a384c-8016-4611-b381-df0695e86d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 14:27:38.187765 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:38.187734 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c02685af-012f-403f-a6cc-e5ad699bb2e2" path="/var/lib/kubelet/pods/c02685af-012f-403f-a6cc-e5ad699bb2e2/volumes" Sep 11 14:27:38.755879 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:38.755824 2573 patch_prober.go:28] interesting pod/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 14:27:38.756249 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:38.755897 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" podUID="3d2a384c-8016-4611-b381-df0695e86d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 14:27:39.755709 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:39.755669 2573 patch_prober.go:28] interesting pod/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 14:27:39.755919 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:39.755731 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" podUID="3d2a384c-8016-4611-b381-df0695e86d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 14:27:40.755513 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:40.755477 2573 patch_prober.go:28] interesting pod/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 14:27:40.755688 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:40.755533 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" podUID="3d2a384c-8016-4611-b381-df0695e86d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 14:27:41.756032 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:41.755995 2573 patch_prober.go:28] interesting pod/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 14:27:41.756425 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:41.756059 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" podUID="3d2a384c-8016-4611-b381-df0695e86d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 14:27:41.963890 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:41.963851 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:42.755673 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:42.755632 2573 patch_prober.go:28] interesting pod/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 14:27:42.755859 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:42.755713 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" podUID="3d2a384c-8016-4611-b381-df0695e86d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 14:27:43.759128 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:43.759094 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:43.760158 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:43.760138 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:27:54.324905 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:54.324872 2573 generic.go:356] "Generic (PLEG): container finished" podID="18fa0554-6c12-40a1-b97c-e82f9a03a6f3" containerID="c5d3be901f6b6b8b2c8d327a94cc43592cc170b4097e894131950c3221a59e5d" exitCode=0 Sep 11 14:27:54.325448 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:54.324952 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" event={"ID":"18fa0554-6c12-40a1-b97c-e82f9a03a6f3","Type":"ContainerDied","Data":"c5d3be901f6b6b8b2c8d327a94cc43592cc170b4097e894131950c3221a59e5d"} Sep 11 14:27:54.325522 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:54.325510 2573 scope.go:117] "RemoveContainer" containerID="c5d3be901f6b6b8b2c8d327a94cc43592cc170b4097e894131950c3221a59e5d" Sep 11 14:27:54.326661 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:54.326620 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:54.326763 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:54.326687 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:54.326763 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:54.326727 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:27:55.337658 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:55.337620 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" event={"ID":"18fa0554-6c12-40a1-b97c-e82f9a03a6f3","Type":"ContainerStarted","Data":"7d61ff337e50476b71de913f4ab5b1affc82fa725a80c0c9329aacbe0dbe1a83"} Sep 11 14:27:56.067544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:56.067507 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:27:56.067728 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:56.067694 2573 patch_prober.go:28] interesting pod/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 11 14:27:56.067774 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:56.067738 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" podUID="18fa0554-6c12-40a1-b97c-e82f9a03a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 11 14:27:57.067654 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:57.067615 2573 patch_prober.go:28] interesting pod/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 11 14:27:57.068112 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:57.067682 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" podUID="18fa0554-6c12-40a1-b97c-e82f9a03a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 11 14:27:58.068450 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:58.068410 2573 patch_prober.go:28] interesting pod/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 11 14:27:58.068873 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:58.068468 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" podUID="18fa0554-6c12-40a1-b97c-e82f9a03a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 11 14:27:59.067901 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:59.067861 2573 patch_prober.go:28] interesting pod/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 11 14:27:59.068088 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:27:59.067919 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" podUID="18fa0554-6c12-40a1-b97c-e82f9a03a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 11 14:28:00.068367 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:00.068330 2573 patch_prober.go:28] interesting pod/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 11 14:28:00.068860 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:00.068388 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" podUID="18fa0554-6c12-40a1-b97c-e82f9a03a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 11 14:28:01.072116 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:01.072084 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:28:01.072486 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:01.072450 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:28:01.073071 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:01.073051 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:28:05.133275 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:05.133234 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p"] Sep 11 14:28:05.133705 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:05.133464 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" podUID="18fa0554-6c12-40a1-b97c-e82f9a03a6f3" containerName="istio-proxy" containerID="cri-o://7d61ff337e50476b71de913f4ab5b1affc82fa725a80c0c9329aacbe0dbe1a83" gracePeriod=30 Sep 11 14:28:06.014266 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.014233 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f"] Sep 11 14:28:06.014920 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.014903 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c02685af-012f-403f-a6cc-e5ad699bb2e2" containerName="istio-proxy" Sep 11 14:28:06.014978 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.014924 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c02685af-012f-403f-a6cc-e5ad699bb2e2" containerName="istio-proxy" Sep 11 14:28:06.014978 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.014945 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c02685af-012f-403f-a6cc-e5ad699bb2e2" containerName="istio-proxy" Sep 11 14:28:06.014978 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.014954 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c02685af-012f-403f-a6cc-e5ad699bb2e2" containerName="istio-proxy" Sep 11 14:28:06.015076 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.015067 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c02685af-012f-403f-a6cc-e5ad699bb2e2" containerName="istio-proxy" Sep 11 14:28:06.015117 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.015081 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c02685af-012f-403f-a6cc-e5ad699bb2e2" containerName="istio-proxy" Sep 11 14:28:06.017759 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.017742 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.021265 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.021244 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f44a68eb-openshift-default-dockercfg-nw9qq\"" Sep 11 14:28:06.034561 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.034539 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f"] Sep 11 14:28:06.125353 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.125321 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-envoy\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.125624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.125603 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-podinfo\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.125797 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.125784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-workload-certs\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.126273 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.125933 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacc1bea-98c4-4e8c-912f-f27220b04db1-istiod-ca-cert\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.126273 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.125998 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-workload-socket\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.126273 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.126044 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-credential-socket\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.126273 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.126078 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-token\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.126273 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.126109 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw42l\" (UniqueName: \"kubernetes.io/projected/aacc1bea-98c4-4e8c-912f-f27220b04db1-kube-api-access-xw42l\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.126273 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.126157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-data\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.227018 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.226979 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-workload-certs\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.227018 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.227013 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacc1bea-98c4-4e8c-912f-f27220b04db1-istiod-ca-cert\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.227544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.227140 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-workload-socket\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.227544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.227209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-credential-socket\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.227544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.227257 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-token\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.227544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.227287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw42l\" (UniqueName: \"kubernetes.io/projected/aacc1bea-98c4-4e8c-912f-f27220b04db1-kube-api-access-xw42l\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.227544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.227354 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-data\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.227544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.227457 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-workload-socket\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.227544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.227480 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-workload-certs\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.227544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.227535 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-envoy\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.228025 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.227562 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-podinfo\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.228025 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.227756 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacc1bea-98c4-4e8c-912f-f27220b04db1-istiod-ca-cert\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.228025 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.227847 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-data\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.228230 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.228201 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-credential-socket\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.229769 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.229742 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-envoy\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.230000 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.229983 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-podinfo\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.236410 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.236386 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw42l\" (UniqueName: \"kubernetes.io/projected/aacc1bea-98c4-4e8c-912f-f27220b04db1-kube-api-access-xw42l\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.236503 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.236461 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-token\") pod \"e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.331043 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.331021 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:06.613234 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.613208 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f"] Sep 11 14:28:06.614666 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:28:06.614642 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacc1bea_98c4_4e8c_912f_f27220b04db1.slice/crio-cde6b8fa9b0e94c49b8d33588cdc2ed4ae2b2e55fcf0ce14061ac0c3dddec629 WatchSource:0}: Error finding container cde6b8fa9b0e94c49b8d33588cdc2ed4ae2b2e55fcf0ce14061ac0c3dddec629: Status 404 returned error can't find the container with id cde6b8fa9b0e94c49b8d33588cdc2ed4ae2b2e55fcf0ce14061ac0c3dddec629 Sep 11 14:28:06.617522 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.617490 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:28:06.617626 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.617553 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:28:06.617626 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:06.617586 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:28:07.421691 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:07.421652 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" event={"ID":"aacc1bea-98c4-4e8c-912f-f27220b04db1","Type":"ContainerStarted","Data":"518f855f20d39fdad21923d10f4aa98a3c559241dbb42c5fa2f8d896bb46128f"} Sep 11 14:28:07.422085 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:07.421740 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" event={"ID":"aacc1bea-98c4-4e8c-912f-f27220b04db1","Type":"ContainerStarted","Data":"cde6b8fa9b0e94c49b8d33588cdc2ed4ae2b2e55fcf0ce14061ac0c3dddec629"} Sep 11 14:28:07.446372 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:07.446315 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" podStartSLOduration=2.4462944589999998 podStartE2EDuration="2.446294459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:07.445401163 +0000 UTC m=+799.782369351" watchObservedRunningTime="2026-09-11 14:28:07.446294459 +0000 UTC m=+799.783262650" Sep 11 14:28:08.331952 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:08.331922 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:08.337022 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:08.336994 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:08.426974 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:08.426945 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:08.427868 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:08.427822 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:10.412211 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.412148 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:28:10.442503 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.442468 2573 generic.go:356] "Generic (PLEG): container finished" podID="18fa0554-6c12-40a1-b97c-e82f9a03a6f3" containerID="7d61ff337e50476b71de913f4ab5b1affc82fa725a80c0c9329aacbe0dbe1a83" exitCode=0 Sep 11 14:28:10.442682 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.442526 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" Sep 11 14:28:10.442682 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.442565 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" event={"ID":"18fa0554-6c12-40a1-b97c-e82f9a03a6f3","Type":"ContainerDied","Data":"7d61ff337e50476b71de913f4ab5b1affc82fa725a80c0c9329aacbe0dbe1a83"} Sep 11 14:28:10.442682 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.442618 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p" event={"ID":"18fa0554-6c12-40a1-b97c-e82f9a03a6f3","Type":"ContainerDied","Data":"15bea6dc4645ee44e850aad24079e0741350866ac39dd992cfaa93035470bcbe"} Sep 11 14:28:10.442682 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.442646 2573 scope.go:117] "RemoveContainer" containerID="7d61ff337e50476b71de913f4ab5b1affc82fa725a80c0c9329aacbe0dbe1a83" Sep 11 14:28:10.454458 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.454426 2573 scope.go:117] "RemoveContainer" containerID="c5d3be901f6b6b8b2c8d327a94cc43592cc170b4097e894131950c3221a59e5d" Sep 11 14:28:10.466003 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.465919 2573 scope.go:117] "RemoveContainer" containerID="7d61ff337e50476b71de913f4ab5b1affc82fa725a80c0c9329aacbe0dbe1a83" Sep 11 14:28:10.466279 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:28:10.466257 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d61ff337e50476b71de913f4ab5b1affc82fa725a80c0c9329aacbe0dbe1a83\": container with ID starting with 7d61ff337e50476b71de913f4ab5b1affc82fa725a80c0c9329aacbe0dbe1a83 not found: ID does not exist" containerID="7d61ff337e50476b71de913f4ab5b1affc82fa725a80c0c9329aacbe0dbe1a83" Sep 11 14:28:10.466331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.466289 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d61ff337e50476b71de913f4ab5b1affc82fa725a80c0c9329aacbe0dbe1a83"} err="failed to get container status \"7d61ff337e50476b71de913f4ab5b1affc82fa725a80c0c9329aacbe0dbe1a83\": rpc error: code = NotFound desc = could not find container \"7d61ff337e50476b71de913f4ab5b1affc82fa725a80c0c9329aacbe0dbe1a83\": container with ID starting with 7d61ff337e50476b71de913f4ab5b1affc82fa725a80c0c9329aacbe0dbe1a83 not found: ID does not exist" Sep 11 14:28:10.466331 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.466308 2573 scope.go:117] "RemoveContainer" containerID="c5d3be901f6b6b8b2c8d327a94cc43592cc170b4097e894131950c3221a59e5d" Sep 11 14:28:10.466618 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:28:10.466600 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d3be901f6b6b8b2c8d327a94cc43592cc170b4097e894131950c3221a59e5d\": container with ID starting with c5d3be901f6b6b8b2c8d327a94cc43592cc170b4097e894131950c3221a59e5d not found: ID does not exist" containerID="c5d3be901f6b6b8b2c8d327a94cc43592cc170b4097e894131950c3221a59e5d" Sep 11 14:28:10.466675 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.466622 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d3be901f6b6b8b2c8d327a94cc43592cc170b4097e894131950c3221a59e5d"} err="failed to get container status \"c5d3be901f6b6b8b2c8d327a94cc43592cc170b4097e894131950c3221a59e5d\": rpc error: code = NotFound desc = could not find container \"c5d3be901f6b6b8b2c8d327a94cc43592cc170b4097e894131950c3221a59e5d\": container with ID starting with c5d3be901f6b6b8b2c8d327a94cc43592cc170b4097e894131950c3221a59e5d not found: ID does not exist" Sep 11 14:28:10.474049 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.474023 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-credential-socket\") pod \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " Sep 11 14:28:10.474151 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.474070 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-token\") pod \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " Sep 11 14:28:10.474151 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.474129 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-data\") pod \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " Sep 11 14:28:10.474247 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.474177 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-podinfo\") pod \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " Sep 11 14:28:10.474247 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.474225 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gh42p\" (UniqueName: \"kubernetes.io/projected/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-kube-api-access-gh42p\") pod \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " Sep 11 14:28:10.474336 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.474267 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18fa0554-6c12-40a1-b97c-e82f9a03a6f3" (UID: "18fa0554-6c12-40a1-b97c-e82f9a03a6f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:28:10.474336 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.474296 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-workload-socket\") pod \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " Sep 11 14:28:10.474418 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.474360 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istiod-ca-cert\") pod \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " Sep 11 14:28:10.474418 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.474405 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-workload-certs\") pod \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " Sep 11 14:28:10.474515 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.474451 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-envoy\") pod \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\" (UID: \"18fa0554-6c12-40a1-b97c-e82f9a03a6f3\") " Sep 11 14:28:10.474515 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.474485 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-data" (OuterVolumeSpecName: "istio-data") pod "18fa0554-6c12-40a1-b97c-e82f9a03a6f3" (UID: "18fa0554-6c12-40a1-b97c-e82f9a03a6f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:28:10.475009 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.474760 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18fa0554-6c12-40a1-b97c-e82f9a03a6f3" (UID: "18fa0554-6c12-40a1-b97c-e82f9a03a6f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:28:10.475009 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.474781 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18fa0554-6c12-40a1-b97c-e82f9a03a6f3" (UID: "18fa0554-6c12-40a1-b97c-e82f9a03a6f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:28:10.475009 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.474827 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:28:10.475009 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.474943 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:28:10.475283 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.475080 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18fa0554-6c12-40a1-b97c-e82f9a03a6f3" (UID: "18fa0554-6c12-40a1-b97c-e82f9a03a6f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:28:10.476586 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.476561 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18fa0554-6c12-40a1-b97c-e82f9a03a6f3" (UID: "18fa0554-6c12-40a1-b97c-e82f9a03a6f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:28:10.476669 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.476568 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-kube-api-access-gh42p" (OuterVolumeSpecName: "kube-api-access-gh42p") pod "18fa0554-6c12-40a1-b97c-e82f9a03a6f3" (UID: "18fa0554-6c12-40a1-b97c-e82f9a03a6f3"). InnerVolumeSpecName "kube-api-access-gh42p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:28:10.476669 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.476587 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-token" (OuterVolumeSpecName: "istio-token") pod "18fa0554-6c12-40a1-b97c-e82f9a03a6f3" (UID: "18fa0554-6c12-40a1-b97c-e82f9a03a6f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:28:10.477114 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.477092 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18fa0554-6c12-40a1-b97c-e82f9a03a6f3" (UID: "18fa0554-6c12-40a1-b97c-e82f9a03a6f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:28:10.576991 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.576745 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:28:10.576991 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.576788 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:28:10.576991 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.576804 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh42p\" (UniqueName: \"kubernetes.io/projected/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-kube-api-access-gh42p\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:28:10.576991 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.576819 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:28:10.576991 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.576889 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:28:10.576991 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.576905 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:28:10.576991 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.576920 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18fa0554-6c12-40a1-b97c-e82f9a03a6f3-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:28:10.773570 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.773533 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p"] Sep 11 14:28:10.778262 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:10.778224 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-2db3d4c3-gw-openshift-default-5f9468dcb4-6ds8p"] Sep 11 14:28:12.188898 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:12.188857 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18fa0554-6c12-40a1-b97c-e82f9a03a6f3" path="/var/lib/kubelet/pods/18fa0554-6c12-40a1-b97c-e82f9a03a6f3/volumes" Sep 11 14:28:28.559936 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:28.559900 2573 generic.go:356] "Generic (PLEG): container finished" podID="aacc1bea-98c4-4e8c-912f-f27220b04db1" containerID="518f855f20d39fdad21923d10f4aa98a3c559241dbb42c5fa2f8d896bb46128f" exitCode=0 Sep 11 14:28:28.560336 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:28.559978 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" event={"ID":"aacc1bea-98c4-4e8c-912f-f27220b04db1","Type":"ContainerDied","Data":"518f855f20d39fdad21923d10f4aa98a3c559241dbb42c5fa2f8d896bb46128f"} Sep 11 14:28:28.560394 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:28.560381 2573 scope.go:117] "RemoveContainer" containerID="518f855f20d39fdad21923d10f4aa98a3c559241dbb42c5fa2f8d896bb46128f" Sep 11 14:28:28.561523 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:28.561497 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:28:28.561584 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:28.561563 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:28:28.561625 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:28.561594 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:28:29.568376 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:29.568334 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" event={"ID":"aacc1bea-98c4-4e8c-912f-f27220b04db1","Type":"ContainerStarted","Data":"6cd67876673f7d195c8c29a27b6d352a69e0f75f38b98cf357ab9a219fa6626f"} Sep 11 14:28:30.331464 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:30.331426 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:30.331738 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:30.331660 2573 patch_prober.go:28] interesting pod/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 14:28:30.331930 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:30.331905 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" podUID="aacc1bea-98c4-4e8c-912f-f27220b04db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 14:28:31.331525 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:31.331481 2573 patch_prober.go:28] interesting pod/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 14:28:31.332043 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:31.331568 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" podUID="aacc1bea-98c4-4e8c-912f-f27220b04db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 14:28:32.332110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:32.332073 2573 patch_prober.go:28] interesting pod/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 14:28:32.332686 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:32.332654 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" podUID="aacc1bea-98c4-4e8c-912f-f27220b04db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 14:28:33.331987 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:33.331950 2573 patch_prober.go:28] interesting pod/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 14:28:33.332290 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:33.332258 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" podUID="aacc1bea-98c4-4e8c-912f-f27220b04db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 14:28:34.331611 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:34.331576 2573 patch_prober.go:28] interesting pod/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 14:28:34.331907 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:34.331878 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" podUID="aacc1bea-98c4-4e8c-912f-f27220b04db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 14:28:35.335394 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:35.335358 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:35.336050 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:35.335651 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:35.336343 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:35.336324 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:28:49.054160 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.054118 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl"] Sep 11 14:28:49.055088 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.055068 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18fa0554-6c12-40a1-b97c-e82f9a03a6f3" containerName="istio-proxy" Sep 11 14:28:49.055088 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.055091 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="18fa0554-6c12-40a1-b97c-e82f9a03a6f3" containerName="istio-proxy" Sep 11 14:28:49.055283 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.055138 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18fa0554-6c12-40a1-b97c-e82f9a03a6f3" containerName="istio-proxy" Sep 11 14:28:49.055283 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.055148 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="18fa0554-6c12-40a1-b97c-e82f9a03a6f3" containerName="istio-proxy" Sep 11 14:28:49.055382 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.055353 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="18fa0554-6c12-40a1-b97c-e82f9a03a6f3" containerName="istio-proxy" Sep 11 14:28:49.055382 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.055372 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="18fa0554-6c12-40a1-b97c-e82f9a03a6f3" containerName="istio-proxy" Sep 11 14:28:49.058697 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.058669 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl" Sep 11 14:28:49.062397 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.062368 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp8gp\"" Sep 11 14:28:49.062518 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.062498 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:28:49.082259 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.082230 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl"] Sep 11 14:28:49.159687 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.159655 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a4371c-b823-4e82-b542-bd94bde07fdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b4a233-dv7zl\" (UID: \"29a4371c-b823-4e82-b542-bd94bde07fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl" Sep 11 14:28:49.261114 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.261076 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a4371c-b823-4e82-b542-bd94bde07fdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b4a233-dv7zl\" (UID: \"29a4371c-b823-4e82-b542-bd94bde07fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl" Sep 11 14:28:49.261702 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.261675 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a4371c-b823-4e82-b542-bd94bde07fdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b4a233-dv7zl\" (UID: \"29a4371c-b823-4e82-b542-bd94bde07fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl" Sep 11 14:28:49.370205 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.370176 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl" Sep 11 14:28:49.645626 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:28:49.645596 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a4371c_b823_4e82_b542_bd94bde07fdb.slice/crio-6f43e2ddf2e538440edeb8f1d164265d6df319beab13e8dbf4b61f05875f409a WatchSource:0}: Error finding container 6f43e2ddf2e538440edeb8f1d164265d6df319beab13e8dbf4b61f05875f409a: Status 404 returned error can't find the container with id 6f43e2ddf2e538440edeb8f1d164265d6df319beab13e8dbf4b61f05875f409a Sep 11 14:28:49.645889 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.645866 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl"] Sep 11 14:28:49.686379 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:49.686352 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl" event={"ID":"29a4371c-b823-4e82-b542-bd94bde07fdb","Type":"ContainerStarted","Data":"6f43e2ddf2e538440edeb8f1d164265d6df319beab13e8dbf4b61f05875f409a"} Sep 11 14:28:52.705629 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:52.705540 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl" event={"ID":"29a4371c-b823-4e82-b542-bd94bde07fdb","Type":"ContainerStarted","Data":"258aa0e3ee6de73003a2777a6d1d61bac387d940847b4c879896489eb4dccf52"} Sep 11 14:28:52.739349 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:52.739298 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl" podStartSLOduration=1.288446686 podStartE2EDuration="3.739282665s" totalImagesPullingTime="2.450835979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:52.737632493 +0000 UTC m=+845.074600683" watchObservedRunningTime="2026-09-11 14:28:52.739282665 +0000 UTC m=+845.076250855" Sep 11 14:28:53.712897 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:53.712863 2573 generic.go:356] "Generic (PLEG): container finished" podID="29a4371c-b823-4e82-b542-bd94bde07fdb" containerID="258aa0e3ee6de73003a2777a6d1d61bac387d940847b4c879896489eb4dccf52" exitCode=0 Sep 11 14:28:53.713312 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:53.712912 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl" event={"ID":"29a4371c-b823-4e82-b542-bd94bde07fdb","Type":"ContainerDied","Data":"258aa0e3ee6de73003a2777a6d1d61bac387d940847b4c879896489eb4dccf52"} Sep 11 14:28:54.845909 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:54.845886 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl" Sep 11 14:28:54.919496 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:54.919465 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a4371c-b823-4e82-b542-bd94bde07fdb-maas-api-service-ca\") pod \"29a4371c-b823-4e82-b542-bd94bde07fdb\" (UID: \"29a4371c-b823-4e82-b542-bd94bde07fdb\") " Sep 11 14:28:54.919817 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:54.919792 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a4371c-b823-4e82-b542-bd94bde07fdb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "29a4371c-b823-4e82-b542-bd94bde07fdb" (UID: "29a4371c-b823-4e82-b542-bd94bde07fdb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:28:55.020813 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:55.020736 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a4371c-b823-4e82-b542-bd94bde07fdb-maas-api-service-ca\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:28:55.260424 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:55.260384 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f44a68eb-798vx"] Sep 11 14:28:55.260826 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:55.260814 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a4371c-b823-4e82-b542-bd94bde07fdb" containerName="revoke-keys" Sep 11 14:28:55.260826 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:55.260828 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a4371c-b823-4e82-b542-bd94bde07fdb" containerName="revoke-keys" Sep 11 14:28:55.260934 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:55.260916 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a4371c-b823-4e82-b542-bd94bde07fdb" containerName="revoke-keys" Sep 11 14:28:55.263129 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:55.263114 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f44a68eb-798vx" Sep 11 14:28:55.272929 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:55.272866 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f44a68eb-798vx"] Sep 11 14:28:55.424432 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:55.424396 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a181fc6-fa43-4a97-9230-01cdbea8c53e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f44a68eb-798vx\" (UID: \"1a181fc6-fa43-4a97-9230-01cdbea8c53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f44a68eb-798vx" Sep 11 14:28:55.525604 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:55.525522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a181fc6-fa43-4a97-9230-01cdbea8c53e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f44a68eb-798vx\" (UID: \"1a181fc6-fa43-4a97-9230-01cdbea8c53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f44a68eb-798vx" Sep 11 14:28:55.526161 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:55.526143 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a181fc6-fa43-4a97-9230-01cdbea8c53e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f44a68eb-798vx\" (UID: \"1a181fc6-fa43-4a97-9230-01cdbea8c53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f44a68eb-798vx" Sep 11 14:28:55.574635 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:55.574607 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f44a68eb-798vx" Sep 11 14:28:55.726139 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:55.726107 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl" Sep 11 14:28:55.726325 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:55.726143 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl" event={"ID":"29a4371c-b823-4e82-b542-bd94bde07fdb","Type":"ContainerDied","Data":"6f43e2ddf2e538440edeb8f1d164265d6df319beab13e8dbf4b61f05875f409a"} Sep 11 14:28:55.726325 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:55.726186 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f43e2ddf2e538440edeb8f1d164265d6df319beab13e8dbf4b61f05875f409a" Sep 11 14:28:55.846930 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:55.846903 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f44a68eb-798vx"] Sep 11 14:28:55.847974 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:28:55.847949 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a181fc6_fa43_4a97_9230_01cdbea8c53e.slice/crio-47c4af17287a0441a80fe35dfaccae9638e193c53d1de960fdc468da92e1acdb WatchSource:0}: Error finding container 47c4af17287a0441a80fe35dfaccae9638e193c53d1de960fdc468da92e1acdb: Status 404 returned error can't find the container with id 47c4af17287a0441a80fe35dfaccae9638e193c53d1de960fdc468da92e1acdb Sep 11 14:28:56.732820 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:56.732786 2573 generic.go:356] "Generic (PLEG): container finished" podID="1a181fc6-fa43-4a97-9230-01cdbea8c53e" containerID="38683571770ac615d8785b70f78a96b48c1416eba95f2835679b1cffbe436bbd" exitCode=0 Sep 11 14:28:56.733017 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:56.732873 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f44a68eb-798vx" event={"ID":"1a181fc6-fa43-4a97-9230-01cdbea8c53e","Type":"ContainerDied","Data":"38683571770ac615d8785b70f78a96b48c1416eba95f2835679b1cffbe436bbd"} Sep 11 14:28:56.733017 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:56.732922 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f44a68eb-798vx" event={"ID":"1a181fc6-fa43-4a97-9230-01cdbea8c53e","Type":"ContainerStarted","Data":"47c4af17287a0441a80fe35dfaccae9638e193c53d1de960fdc468da92e1acdb"} Sep 11 14:28:57.870420 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:57.870398 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f44a68eb-798vx" Sep 11 14:28:57.950747 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:57.950715 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a181fc6-fa43-4a97-9230-01cdbea8c53e-maas-api-service-ca\") pod \"1a181fc6-fa43-4a97-9230-01cdbea8c53e\" (UID: \"1a181fc6-fa43-4a97-9230-01cdbea8c53e\") " Sep 11 14:28:57.951099 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:57.951077 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a181fc6-fa43-4a97-9230-01cdbea8c53e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1a181fc6-fa43-4a97-9230-01cdbea8c53e" (UID: "1a181fc6-fa43-4a97-9230-01cdbea8c53e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:28:58.052030 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.051937 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a181fc6-fa43-4a97-9230-01cdbea8c53e-maas-api-service-ca\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:28:58.599873 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.599817 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc"] Sep 11 14:28:58.600269 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.600256 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a181fc6-fa43-4a97-9230-01cdbea8c53e" containerName="revoke-keys" Sep 11 14:28:58.600320 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.600271 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a181fc6-fa43-4a97-9230-01cdbea8c53e" containerName="revoke-keys" Sep 11 14:28:58.601043 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.601015 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a181fc6-fa43-4a97-9230-01cdbea8c53e" containerName="revoke-keys" Sep 11 14:28:58.609511 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.609480 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.617980 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.617959 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a39864\"/\"default-dockercfg-kdrb7\"" Sep 11 14:28:58.621543 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.621519 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a39864\"/\"test-model-a39864-kserve-self-signed-certs\"" Sep 11 14:28:58.621662 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.621613 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a39864\"/\"openshift-service-ca.crt\"" Sep 11 14:28:58.621662 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.621623 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a39864\"/\"kube-root-ca.crt\"" Sep 11 14:28:58.717692 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.717655 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc"] Sep 11 14:28:58.753481 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.753454 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f44a68eb-798vx" Sep 11 14:28:58.753798 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.753514 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f44a68eb-798vx" event={"ID":"1a181fc6-fa43-4a97-9230-01cdbea8c53e","Type":"ContainerDied","Data":"47c4af17287a0441a80fe35dfaccae9638e193c53d1de960fdc468da92e1acdb"} Sep 11 14:28:58.753920 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.753853 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47c4af17287a0441a80fe35dfaccae9638e193c53d1de960fdc468da92e1acdb" Sep 11 14:28:58.758609 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.758561 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zbfp\" (UniqueName: \"kubernetes.io/projected/12f35580-a375-449d-a57b-3bee55c487dd-kube-api-access-9zbfp\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.758915 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.758892 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-model-cache\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.759416 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.759367 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f35580-a375-449d-a57b-3bee55c487dd-tls-certs\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.759646 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.759629 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-home\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.759812 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.759796 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-dshm\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.760028 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.759989 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-tmp-dir\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.861847 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.861750 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-model-cache\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.861847 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.861803 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f35580-a375-449d-a57b-3bee55c487dd-tls-certs\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.862077 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.861848 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-home\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.862077 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.861876 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-dshm\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.862077 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.861901 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-tmp-dir\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.862077 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.862059 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zbfp\" (UniqueName: \"kubernetes.io/projected/12f35580-a375-449d-a57b-3bee55c487dd-kube-api-access-9zbfp\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.862285 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.862228 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-model-cache\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.862342 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.862295 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-home\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.862397 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.862306 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-tmp-dir\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.864148 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.864125 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-dshm\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.864568 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.864548 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f35580-a375-449d-a57b-3bee55c487dd-tls-certs\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.871798 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.871774 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zbfp\" (UniqueName: \"kubernetes.io/projected/12f35580-a375-449d-a57b-3bee55c487dd-kube-api-access-9zbfp\") pod \"test-model-a39864-kserve-655b77cb9f-kqqxc\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:58.919770 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:58.919736 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:28:59.203536 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:59.203455 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc"] Sep 11 14:28:59.207600 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:28:59.207568 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12f35580_a375_449d_a57b_3bee55c487dd.slice/crio-6906459a0f5cfa6d799cfe3e44fc6a862b7b0966e9032401ed895c64509d019b WatchSource:0}: Error finding container 6906459a0f5cfa6d799cfe3e44fc6a862b7b0966e9032401ed895c64509d019b: Status 404 returned error can't find the container with id 6906459a0f5cfa6d799cfe3e44fc6a862b7b0966e9032401ed895c64509d019b Sep 11 14:28:59.761871 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:59.761819 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" event={"ID":"12f35580-a375-449d-a57b-3bee55c487dd","Type":"ContainerStarted","Data":"56cb37237b79b903cd3e466e377b4bcf433732e1b543c6fb2611f5d3d433ea57"} Sep 11 14:28:59.761871 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:59.761875 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" event={"ID":"12f35580-a375-449d-a57b-3bee55c487dd","Type":"ContainerStarted","Data":"6906459a0f5cfa6d799cfe3e44fc6a862b7b0966e9032401ed895c64509d019b"} Sep 11 14:28:59.804850 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:28:59.804777 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" podStartSLOduration=1.8047616180000001 podStartE2EDuration="1.804761618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:59.802637822 +0000 UTC m=+852.139606013" watchObservedRunningTime="2026-09-11 14:28:59.804761618 +0000 UTC m=+852.141729866" Sep 11 14:29:00.293952 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:00.293910 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl"] Sep 11 14:29:00.301093 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:00.301056 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b4a233-dv7zl"] Sep 11 14:29:02.187743 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:02.187712 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29a4371c-b823-4e82-b542-bd94bde07fdb" path="/var/lib/kubelet/pods/29a4371c-b823-4e82-b542-bd94bde07fdb/volumes" Sep 11 14:29:06.118009 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:06.117969 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f44a68eb-798vx"] Sep 11 14:29:06.120569 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:06.120545 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f44a68eb-798vx"] Sep 11 14:29:06.187696 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:06.187663 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a181fc6-fa43-4a97-9230-01cdbea8c53e" path="/var/lib/kubelet/pods/1a181fc6-fa43-4a97-9230-01cdbea8c53e/volumes" Sep 11 14:29:07.342273 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:07.342235 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f"] Sep 11 14:29:07.342689 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:07.342480 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" podUID="aacc1bea-98c4-4e8c-912f-f27220b04db1" containerName="istio-proxy" containerID="cri-o://6cd67876673f7d195c8c29a27b6d352a69e0f75f38b98cf357ab9a219fa6626f" gracePeriod=30 Sep 11 14:29:08.221335 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.221302 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw"] Sep 11 14:29:08.226409 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.226392 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.229767 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.229744 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b0010444-openshift-default-dockercfg-w7z8m\"" Sep 11 14:29:08.240957 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.240933 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw"] Sep 11 14:29:08.261919 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.261890 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-token\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.262044 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.261974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfpkm\" (UniqueName: \"kubernetes.io/projected/a8c70903-ef19-4ef3-894a-ece8736d5037-kube-api-access-sfpkm\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.262044 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.262023 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-data\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.262138 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.262066 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-podinfo\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.262138 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.262095 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-workload-certs\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.262138 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.262114 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-workload-socket\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.262278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.262255 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-credential-socket\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.262332 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.262315 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c70903-ef19-4ef3-894a-ece8736d5037-istiod-ca-cert\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.262369 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.262358 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-envoy\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.363128 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.363094 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfpkm\" (UniqueName: \"kubernetes.io/projected/a8c70903-ef19-4ef3-894a-ece8736d5037-kube-api-access-sfpkm\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.363128 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.363131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-data\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.363638 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.363156 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-podinfo\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.363638 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.363268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-workload-certs\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.363638 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.363312 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-workload-socket\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.363638 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.363409 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-credential-socket\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.363638 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.363470 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c70903-ef19-4ef3-894a-ece8736d5037-istiod-ca-cert\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.363638 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.363516 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-envoy\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.363638 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.363560 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-data\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.363638 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.363609 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-token\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.363638 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.363619 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-workload-certs\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.364111 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.363699 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-workload-socket\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.364111 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.363770 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-credential-socket\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.364241 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.364224 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c70903-ef19-4ef3-894a-ece8736d5037-istiod-ca-cert\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.366032 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.366012 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-envoy\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.366341 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.366318 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-podinfo\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.372088 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.372065 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-token\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.372198 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.372087 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfpkm\" (UniqueName: \"kubernetes.io/projected/a8c70903-ef19-4ef3-894a-ece8736d5037-kube-api-access-sfpkm\") pod \"e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.539819 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.538047 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:08.823021 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.822989 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw"] Sep 11 14:29:08.823849 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:29:08.823806 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c70903_ef19_4ef3_894a_ece8736d5037.slice/crio-2c9344eb5dbdae703f304f0eca3fb0fe5d7ec549a603a7fd82d5bdaa72144775 WatchSource:0}: Error finding container 2c9344eb5dbdae703f304f0eca3fb0fe5d7ec549a603a7fd82d5bdaa72144775: Status 404 returned error can't find the container with id 2c9344eb5dbdae703f304f0eca3fb0fe5d7ec549a603a7fd82d5bdaa72144775 Sep 11 14:29:08.826133 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.826101 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:29:08.826212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.826163 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:29:08.826212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.826192 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:29:08.920236 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.920197 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:29:08.920397 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.920256 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:29:08.939228 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:08.939201 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:29:09.823091 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:09.823051 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" event={"ID":"a8c70903-ef19-4ef3-894a-ece8736d5037","Type":"ContainerStarted","Data":"3f70156eead7101a41c0f552d644956b61c5f79cbe58402ac80e9ccec07b055f"} Sep 11 14:29:09.823579 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:09.823115 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" event={"ID":"a8c70903-ef19-4ef3-894a-ece8736d5037","Type":"ContainerStarted","Data":"2c9344eb5dbdae703f304f0eca3fb0fe5d7ec549a603a7fd82d5bdaa72144775"} Sep 11 14:29:09.836588 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:09.836556 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:29:09.931709 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:09.931642 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" podStartSLOduration=1.931624856 podStartE2EDuration="1.931624856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:29:09.876894516 +0000 UTC m=+862.213862706" watchObservedRunningTime="2026-09-11 14:29:09.931624856 +0000 UTC m=+862.268593047" Sep 11 14:29:10.538938 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:10.538891 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:10.544462 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:10.544438 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:10.829436 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:10.829335 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:10.830280 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:10.830256 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:12.598182 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.598159 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:29:12.706617 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.706525 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-podinfo\") pod \"aacc1bea-98c4-4e8c-912f-f27220b04db1\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " Sep 11 14:29:12.706617 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.706569 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-workload-socket\") pod \"aacc1bea-98c4-4e8c-912f-f27220b04db1\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " Sep 11 14:29:12.706617 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.706594 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacc1bea-98c4-4e8c-912f-f27220b04db1-istiod-ca-cert\") pod \"aacc1bea-98c4-4e8c-912f-f27220b04db1\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " Sep 11 14:29:12.706941 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.706638 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xw42l\" (UniqueName: \"kubernetes.io/projected/aacc1bea-98c4-4e8c-912f-f27220b04db1-kube-api-access-xw42l\") pod \"aacc1bea-98c4-4e8c-912f-f27220b04db1\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " Sep 11 14:29:12.706941 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.706668 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-data\") pod \"aacc1bea-98c4-4e8c-912f-f27220b04db1\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " Sep 11 14:29:12.706941 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.706712 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-envoy\") pod \"aacc1bea-98c4-4e8c-912f-f27220b04db1\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " Sep 11 14:29:12.706941 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.706751 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-token\") pod \"aacc1bea-98c4-4e8c-912f-f27220b04db1\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " Sep 11 14:29:12.706941 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.706815 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-credential-socket\") pod \"aacc1bea-98c4-4e8c-912f-f27220b04db1\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " Sep 11 14:29:12.706941 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.706854 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aacc1bea-98c4-4e8c-912f-f27220b04db1" (UID: "aacc1bea-98c4-4e8c-912f-f27220b04db1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:29:12.706941 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.706900 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-workload-certs\") pod \"aacc1bea-98c4-4e8c-912f-f27220b04db1\" (UID: \"aacc1bea-98c4-4e8c-912f-f27220b04db1\") " Sep 11 14:29:12.707301 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.707036 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-data" (OuterVolumeSpecName: "istio-data") pod "aacc1bea-98c4-4e8c-912f-f27220b04db1" (UID: "aacc1bea-98c4-4e8c-912f-f27220b04db1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:29:12.707301 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.707113 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacc1bea-98c4-4e8c-912f-f27220b04db1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aacc1bea-98c4-4e8c-912f-f27220b04db1" (UID: "aacc1bea-98c4-4e8c-912f-f27220b04db1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:29:12.707301 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.707235 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aacc1bea-98c4-4e8c-912f-f27220b04db1" (UID: "aacc1bea-98c4-4e8c-912f-f27220b04db1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:29:12.707460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.707350 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:12.707460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.707370 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacc1bea-98c4-4e8c-912f-f27220b04db1-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:12.707460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.707387 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:12.707460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.707412 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:12.707460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.707367 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aacc1bea-98c4-4e8c-912f-f27220b04db1" (UID: "aacc1bea-98c4-4e8c-912f-f27220b04db1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:29:12.709016 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.708979 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aacc1bea-98c4-4e8c-912f-f27220b04db1" (UID: "aacc1bea-98c4-4e8c-912f-f27220b04db1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:29:12.709131 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.708985 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aacc1bea-98c4-4e8c-912f-f27220b04db1" (UID: "aacc1bea-98c4-4e8c-912f-f27220b04db1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:29:12.709131 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.709046 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-token" (OuterVolumeSpecName: "istio-token") pod "aacc1bea-98c4-4e8c-912f-f27220b04db1" (UID: "aacc1bea-98c4-4e8c-912f-f27220b04db1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:29:12.709322 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.709297 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacc1bea-98c4-4e8c-912f-f27220b04db1-kube-api-access-xw42l" (OuterVolumeSpecName: "kube-api-access-xw42l") pod "aacc1bea-98c4-4e8c-912f-f27220b04db1" (UID: "aacc1bea-98c4-4e8c-912f-f27220b04db1"). InnerVolumeSpecName "kube-api-access-xw42l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:29:12.808444 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.808409 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:12.808444 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.808436 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw42l\" (UniqueName: \"kubernetes.io/projected/aacc1bea-98c4-4e8c-912f-f27220b04db1-kube-api-access-xw42l\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:12.808444 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.808446 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:12.808444 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.808454 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacc1bea-98c4-4e8c-912f-f27220b04db1-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:12.808705 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.808461 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacc1bea-98c4-4e8c-912f-f27220b04db1-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:12.845447 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.845411 2573 generic.go:356] "Generic (PLEG): container finished" podID="aacc1bea-98c4-4e8c-912f-f27220b04db1" containerID="6cd67876673f7d195c8c29a27b6d352a69e0f75f38b98cf357ab9a219fa6626f" exitCode=0 Sep 11 14:29:12.845585 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.845462 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" Sep 11 14:29:12.845585 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.845488 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" event={"ID":"aacc1bea-98c4-4e8c-912f-f27220b04db1","Type":"ContainerDied","Data":"6cd67876673f7d195c8c29a27b6d352a69e0f75f38b98cf357ab9a219fa6626f"} Sep 11 14:29:12.845585 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.845537 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f" event={"ID":"aacc1bea-98c4-4e8c-912f-f27220b04db1","Type":"ContainerDied","Data":"cde6b8fa9b0e94c49b8d33588cdc2ed4ae2b2e55fcf0ce14061ac0c3dddec629"} Sep 11 14:29:12.845585 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.845562 2573 scope.go:117] "RemoveContainer" containerID="6cd67876673f7d195c8c29a27b6d352a69e0f75f38b98cf357ab9a219fa6626f" Sep 11 14:29:12.855588 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.855570 2573 scope.go:117] "RemoveContainer" containerID="518f855f20d39fdad21923d10f4aa98a3c559241dbb42c5fa2f8d896bb46128f" Sep 11 14:29:12.864145 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.864128 2573 scope.go:117] "RemoveContainer" containerID="6cd67876673f7d195c8c29a27b6d352a69e0f75f38b98cf357ab9a219fa6626f" Sep 11 14:29:12.864401 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:29:12.864381 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd67876673f7d195c8c29a27b6d352a69e0f75f38b98cf357ab9a219fa6626f\": container with ID starting with 6cd67876673f7d195c8c29a27b6d352a69e0f75f38b98cf357ab9a219fa6626f not found: ID does not exist" containerID="6cd67876673f7d195c8c29a27b6d352a69e0f75f38b98cf357ab9a219fa6626f" Sep 11 14:29:12.864473 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.864407 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd67876673f7d195c8c29a27b6d352a69e0f75f38b98cf357ab9a219fa6626f"} err="failed to get container status \"6cd67876673f7d195c8c29a27b6d352a69e0f75f38b98cf357ab9a219fa6626f\": rpc error: code = NotFound desc = could not find container \"6cd67876673f7d195c8c29a27b6d352a69e0f75f38b98cf357ab9a219fa6626f\": container with ID starting with 6cd67876673f7d195c8c29a27b6d352a69e0f75f38b98cf357ab9a219fa6626f not found: ID does not exist" Sep 11 14:29:12.864473 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.864422 2573 scope.go:117] "RemoveContainer" containerID="518f855f20d39fdad21923d10f4aa98a3c559241dbb42c5fa2f8d896bb46128f" Sep 11 14:29:12.864631 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:29:12.864614 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518f855f20d39fdad21923d10f4aa98a3c559241dbb42c5fa2f8d896bb46128f\": container with ID starting with 518f855f20d39fdad21923d10f4aa98a3c559241dbb42c5fa2f8d896bb46128f not found: ID does not exist" containerID="518f855f20d39fdad21923d10f4aa98a3c559241dbb42c5fa2f8d896bb46128f" Sep 11 14:29:12.864667 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.864636 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518f855f20d39fdad21923d10f4aa98a3c559241dbb42c5fa2f8d896bb46128f"} err="failed to get container status \"518f855f20d39fdad21923d10f4aa98a3c559241dbb42c5fa2f8d896bb46128f\": rpc error: code = NotFound desc = could not find container \"518f855f20d39fdad21923d10f4aa98a3c559241dbb42c5fa2f8d896bb46128f\": container with ID starting with 518f855f20d39fdad21923d10f4aa98a3c559241dbb42c5fa2f8d896bb46128f not found: ID does not exist" Sep 11 14:29:12.913144 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.913104 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f"] Sep 11 14:29:12.915584 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:12.915564 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f44a68eb-openshift-default-6c7fb489fb-nrc6f"] Sep 11 14:29:14.188618 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:14.188585 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aacc1bea-98c4-4e8c-912f-f27220b04db1" path="/var/lib/kubelet/pods/aacc1bea-98c4-4e8c-912f-f27220b04db1/volumes" Sep 11 14:29:29.949468 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:29.949431 2573 generic.go:356] "Generic (PLEG): container finished" podID="a8c70903-ef19-4ef3-894a-ece8736d5037" containerID="3f70156eead7101a41c0f552d644956b61c5f79cbe58402ac80e9ccec07b055f" exitCode=0 Sep 11 14:29:29.949911 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:29.949501 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" event={"ID":"a8c70903-ef19-4ef3-894a-ece8736d5037","Type":"ContainerDied","Data":"3f70156eead7101a41c0f552d644956b61c5f79cbe58402ac80e9ccec07b055f"} Sep 11 14:29:29.949961 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:29.949953 2573 scope.go:117] "RemoveContainer" containerID="3f70156eead7101a41c0f552d644956b61c5f79cbe58402ac80e9ccec07b055f" Sep 11 14:29:29.951268 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:29.951237 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:29:29.951339 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:29.951311 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:29:29.951377 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:29.951344 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:29:30.959356 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:30.959322 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" event={"ID":"a8c70903-ef19-4ef3-894a-ece8736d5037","Type":"ContainerStarted","Data":"763aa4e842f3a9aa3d3aa0a18ee99f7b8d7c17bad6799bbb52c80c37f7224d77"} Sep 11 14:29:31.539281 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:31.539245 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:31.539603 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:31.539580 2573 patch_prober.go:28] interesting pod/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 14:29:31.539785 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:31.539760 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" podUID="a8c70903-ef19-4ef3-894a-ece8736d5037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 14:29:32.539529 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:32.539491 2573 patch_prober.go:28] interesting pod/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 14:29:32.540101 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:32.540067 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" podUID="a8c70903-ef19-4ef3-894a-ece8736d5037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 14:29:33.539198 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:33.539160 2573 patch_prober.go:28] interesting pod/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 14:29:33.539496 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:33.539463 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" podUID="a8c70903-ef19-4ef3-894a-ece8736d5037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 14:29:34.539408 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:34.539352 2573 patch_prober.go:28] interesting pod/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 14:29:34.539916 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:34.539431 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" podUID="a8c70903-ef19-4ef3-894a-ece8736d5037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 14:29:35.539350 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:35.539309 2573 patch_prober.go:28] interesting pod/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 14:29:35.539584 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:35.539385 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" podUID="a8c70903-ef19-4ef3-894a-ece8736d5037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 14:29:36.543292 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:36.543261 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:36.543700 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:36.543531 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:36.544266 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:36.544249 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:44.218633 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:44.218550 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz"] Sep 11 14:29:44.219169 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:44.218810 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" podUID="2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" containerName="istio-proxy" containerID="cri-o://d5b7dd6b701b6080ce3549c7201e96c1911cd411c0049e0da3b1de7776c47a35" gracePeriod=30 Sep 11 14:29:45.101043 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.101007 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t"] Sep 11 14:29:45.101442 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.101428 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aacc1bea-98c4-4e8c-912f-f27220b04db1" containerName="istio-proxy" Sep 11 14:29:45.101496 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.101444 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aacc1bea-98c4-4e8c-912f-f27220b04db1" containerName="istio-proxy" Sep 11 14:29:45.101550 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.101541 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aacc1bea-98c4-4e8c-912f-f27220b04db1" containerName="istio-proxy" Sep 11 14:29:45.103821 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.103804 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.107774 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.107752 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-696fee-openshift-default-dockercfg-p2z96\"" Sep 11 14:29:45.129827 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.129407 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t"] Sep 11 14:29:45.146010 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.145979 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-envoy\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.146146 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.146071 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-data\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.146146 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.146106 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-credential-socket\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.146271 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.146142 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-podinfo\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.146271 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.146212 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-workload-socket\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.146271 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.146240 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-workload-certs\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.146423 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.146274 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istiod-ca-cert\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.146423 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.146322 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-token\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.146423 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.146348 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-497zm\" (UniqueName: \"kubernetes.io/projected/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-kube-api-access-497zm\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.247518 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.247475 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-envoy\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.248146 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.248120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-data\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.248284 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.248268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-credential-socket\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.248394 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.248380 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-podinfo\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.248556 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.248542 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-workload-socket\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.248670 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.248656 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-workload-certs\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.248784 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.248769 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istiod-ca-cert\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.248936 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.248917 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-token\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.249066 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.249050 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-497zm\" (UniqueName: \"kubernetes.io/projected/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-kube-api-access-497zm\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.252015 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.250629 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istiod-ca-cert\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.252015 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.250912 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-workload-certs\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.252015 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.251405 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-credential-socket\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.252015 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.251649 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-data\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.252579 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.252555 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-envoy\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.252997 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.252975 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-workload-socket\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.255041 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.255022 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-podinfo\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.262425 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.262402 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-497zm\" (UniqueName: \"kubernetes.io/projected/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-kube-api-access-497zm\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.272286 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.272260 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-token\") pod \"e2e-explicit-696fee-openshift-default-848b84776-vwd7t\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.415135 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.415050 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:45.737949 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.737922 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t"] Sep 11 14:29:45.739566 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:29:45.739540 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630cd4b9_d48c_4605_be3c_b2f1eedc70ef.slice/crio-7e86c808a6aeacbed6c83d32c2eed9fb954a32f74bb63ed9c424bb6c061145ec WatchSource:0}: Error finding container 7e86c808a6aeacbed6c83d32c2eed9fb954a32f74bb63ed9c424bb6c061145ec: Status 404 returned error can't find the container with id 7e86c808a6aeacbed6c83d32c2eed9fb954a32f74bb63ed9c424bb6c061145ec Sep 11 14:29:45.741659 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.741628 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:29:45.741723 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.741689 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:29:45.741723 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:45.741718 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:29:46.049782 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:46.049682 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" event={"ID":"630cd4b9-d48c-4605-be3c-b2f1eedc70ef","Type":"ContainerStarted","Data":"94eb2a29465bc4fea502c4d41acbbd63b6b5d2aca85868194588086e5462ecd2"} Sep 11 14:29:46.049782 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:46.049744 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" event={"ID":"630cd4b9-d48c-4605-be3c-b2f1eedc70ef","Type":"ContainerStarted","Data":"7e86c808a6aeacbed6c83d32c2eed9fb954a32f74bb63ed9c424bb6c061145ec"} Sep 11 14:29:46.081216 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:46.081171 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" podStartSLOduration=1.081157364 podStartE2EDuration="1.081157364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:29:46.078993696 +0000 UTC m=+898.415961887" watchObservedRunningTime="2026-09-11 14:29:46.081157364 +0000 UTC m=+898.418125555" Sep 11 14:29:46.415447 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:46.415409 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:46.420691 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:46.420664 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:47.056338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:47.056308 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:47.057216 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:47.057200 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:29:49.473072 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.473046 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:29:49.594304 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.594276 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istiod-ca-cert\") pod \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " Sep 11 14:29:49.594453 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.594312 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-workload-socket\") pod \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " Sep 11 14:29:49.594453 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.594351 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-data\") pod \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " Sep 11 14:29:49.594453 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.594368 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-workload-certs\") pod \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " Sep 11 14:29:49.594453 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.594386 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-credential-socket\") pod \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " Sep 11 14:29:49.594453 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.594410 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-podinfo\") pod \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " Sep 11 14:29:49.594690 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.594501 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxhq8\" (UniqueName: \"kubernetes.io/projected/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-kube-api-access-sxhq8\") pod \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " Sep 11 14:29:49.594690 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.594546 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-envoy\") pod \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " Sep 11 14:29:49.594690 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.594573 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-token\") pod \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\" (UID: \"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d\") " Sep 11 14:29:49.594874 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.594689 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" (UID: "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:29:49.594874 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.594710 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-data" (OuterVolumeSpecName: "istio-data") pod "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" (UID: "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:29:49.594874 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.594728 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" (UID: "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:29:49.594874 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.594752 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" (UID: "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:29:49.595083 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.594951 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" (UID: "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:29:49.595139 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.595094 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:49.595139 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.595114 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:49.595139 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.595123 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:49.595139 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.595131 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:49.595139 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.595139 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:49.596793 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.596760 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" (UID: "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:29:49.596926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.596852 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" (UID: "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:29:49.597185 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.597167 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-token" (OuterVolumeSpecName: "istio-token") pod "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" (UID: "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:29:49.597236 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.597172 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-kube-api-access-sxhq8" (OuterVolumeSpecName: "kube-api-access-sxhq8") pod "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" (UID: "2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d"). InnerVolumeSpecName "kube-api-access-sxhq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:29:49.696369 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.696332 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:49.696369 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.696363 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxhq8\" (UniqueName: \"kubernetes.io/projected/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-kube-api-access-sxhq8\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:49.696369 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.696373 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:49.696594 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:49.696381 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:50.079419 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:50.079331 2573 generic.go:356] "Generic (PLEG): container finished" podID="2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" containerID="d5b7dd6b701b6080ce3549c7201e96c1911cd411c0049e0da3b1de7776c47a35" exitCode=0 Sep 11 14:29:50.079419 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:50.079381 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" Sep 11 14:29:50.079419 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:50.079402 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" event={"ID":"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d","Type":"ContainerDied","Data":"d5b7dd6b701b6080ce3549c7201e96c1911cd411c0049e0da3b1de7776c47a35"} Sep 11 14:29:50.079660 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:50.079473 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz" event={"ID":"2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d","Type":"ContainerDied","Data":"426c5b5fce7221f278e9374e66951e916edc49c2c9e0cc9de2ac112216cbce28"} Sep 11 14:29:50.079660 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:50.079494 2573 scope.go:117] "RemoveContainer" containerID="d5b7dd6b701b6080ce3549c7201e96c1911cd411c0049e0da3b1de7776c47a35" Sep 11 14:29:50.089788 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:50.089770 2573 scope.go:117] "RemoveContainer" containerID="cc2da33b2ef3736e9361040c67b44b61d49257f0ef3160967c5e2b0bf8435f5e" Sep 11 14:29:50.098637 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:50.098617 2573 scope.go:117] "RemoveContainer" containerID="d5b7dd6b701b6080ce3549c7201e96c1911cd411c0049e0da3b1de7776c47a35" Sep 11 14:29:50.098920 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:29:50.098902 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b7dd6b701b6080ce3549c7201e96c1911cd411c0049e0da3b1de7776c47a35\": container with ID starting with d5b7dd6b701b6080ce3549c7201e96c1911cd411c0049e0da3b1de7776c47a35 not found: ID does not exist" containerID="d5b7dd6b701b6080ce3549c7201e96c1911cd411c0049e0da3b1de7776c47a35" Sep 11 14:29:50.098974 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:50.098929 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b7dd6b701b6080ce3549c7201e96c1911cd411c0049e0da3b1de7776c47a35"} err="failed to get container status \"d5b7dd6b701b6080ce3549c7201e96c1911cd411c0049e0da3b1de7776c47a35\": rpc error: code = NotFound desc = could not find container \"d5b7dd6b701b6080ce3549c7201e96c1911cd411c0049e0da3b1de7776c47a35\": container with ID starting with d5b7dd6b701b6080ce3549c7201e96c1911cd411c0049e0da3b1de7776c47a35 not found: ID does not exist" Sep 11 14:29:50.098974 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:50.098947 2573 scope.go:117] "RemoveContainer" containerID="cc2da33b2ef3736e9361040c67b44b61d49257f0ef3160967c5e2b0bf8435f5e" Sep 11 14:29:50.099202 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:29:50.099183 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2da33b2ef3736e9361040c67b44b61d49257f0ef3160967c5e2b0bf8435f5e\": container with ID starting with cc2da33b2ef3736e9361040c67b44b61d49257f0ef3160967c5e2b0bf8435f5e not found: ID does not exist" containerID="cc2da33b2ef3736e9361040c67b44b61d49257f0ef3160967c5e2b0bf8435f5e" Sep 11 14:29:50.099243 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:50.099209 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2da33b2ef3736e9361040c67b44b61d49257f0ef3160967c5e2b0bf8435f5e"} err="failed to get container status \"cc2da33b2ef3736e9361040c67b44b61d49257f0ef3160967c5e2b0bf8435f5e\": rpc error: code = NotFound desc = could not find container \"cc2da33b2ef3736e9361040c67b44b61d49257f0ef3160967c5e2b0bf8435f5e\": container with ID starting with cc2da33b2ef3736e9361040c67b44b61d49257f0ef3160967c5e2b0bf8435f5e not found: ID does not exist" Sep 11 14:29:50.117555 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:50.117527 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz"] Sep 11 14:29:50.120116 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:50.120095 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-b4a233-openshift-default-85f6859cc6-44cbz"] Sep 11 14:29:50.188251 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:50.188222 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" path="/var/lib/kubelet/pods/2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d/volumes" Sep 11 14:29:53.714127 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:53.714095 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw"] Sep 11 14:29:53.714506 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:53.714303 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" podUID="a8c70903-ef19-4ef3-894a-ece8736d5037" containerName="istio-proxy" containerID="cri-o://763aa4e842f3a9aa3d3aa0a18ee99f7b8d7c17bad6799bbb52c80c37f7224d77" gracePeriod=30 Sep 11 14:29:58.971423 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:58.971394 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:59.097413 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.097373 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-workload-socket\") pod \"a8c70903-ef19-4ef3-894a-ece8736d5037\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " Sep 11 14:29:59.097587 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.097455 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c70903-ef19-4ef3-894a-ece8736d5037-istiod-ca-cert\") pod \"a8c70903-ef19-4ef3-894a-ece8736d5037\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " Sep 11 14:29:59.097587 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.097503 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-credential-socket\") pod \"a8c70903-ef19-4ef3-894a-ece8736d5037\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " Sep 11 14:29:59.097587 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.097544 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-data\") pod \"a8c70903-ef19-4ef3-894a-ece8736d5037\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " Sep 11 14:29:59.097756 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.097596 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-podinfo\") pod \"a8c70903-ef19-4ef3-894a-ece8736d5037\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " Sep 11 14:29:59.097756 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.097628 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-workload-certs\") pod \"a8c70903-ef19-4ef3-894a-ece8736d5037\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " Sep 11 14:29:59.097756 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.097656 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-envoy\") pod \"a8c70903-ef19-4ef3-894a-ece8736d5037\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " Sep 11 14:29:59.097756 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.097661 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a8c70903-ef19-4ef3-894a-ece8736d5037" (UID: "a8c70903-ef19-4ef3-894a-ece8736d5037"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:29:59.097756 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.097685 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-token\") pod \"a8c70903-ef19-4ef3-894a-ece8736d5037\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " Sep 11 14:29:59.097756 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.097727 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfpkm\" (UniqueName: \"kubernetes.io/projected/a8c70903-ef19-4ef3-894a-ece8736d5037-kube-api-access-sfpkm\") pod \"a8c70903-ef19-4ef3-894a-ece8736d5037\" (UID: \"a8c70903-ef19-4ef3-894a-ece8736d5037\") " Sep 11 14:29:59.098065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.097793 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a8c70903-ef19-4ef3-894a-ece8736d5037" (UID: "a8c70903-ef19-4ef3-894a-ece8736d5037"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:29:59.098065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.097805 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-data" (OuterVolumeSpecName: "istio-data") pod "a8c70903-ef19-4ef3-894a-ece8736d5037" (UID: "a8c70903-ef19-4ef3-894a-ece8736d5037"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:29:59.098065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.097868 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c70903-ef19-4ef3-894a-ece8736d5037-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a8c70903-ef19-4ef3-894a-ece8736d5037" (UID: "a8c70903-ef19-4ef3-894a-ece8736d5037"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:29:59.098065 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.098013 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a8c70903-ef19-4ef3-894a-ece8736d5037" (UID: "a8c70903-ef19-4ef3-894a-ece8736d5037"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:29:59.098280 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.098180 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:59.098280 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.098201 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c70903-ef19-4ef3-894a-ece8736d5037-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:59.098280 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.098214 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:59.098280 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.098226 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:59.098280 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.098238 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:59.099825 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.099794 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a8c70903-ef19-4ef3-894a-ece8736d5037" (UID: "a8c70903-ef19-4ef3-894a-ece8736d5037"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:29:59.099963 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.099800 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a8c70903-ef19-4ef3-894a-ece8736d5037" (UID: "a8c70903-ef19-4ef3-894a-ece8736d5037"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:29:59.100095 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.100079 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c70903-ef19-4ef3-894a-ece8736d5037-kube-api-access-sfpkm" (OuterVolumeSpecName: "kube-api-access-sfpkm") pod "a8c70903-ef19-4ef3-894a-ece8736d5037" (UID: "a8c70903-ef19-4ef3-894a-ece8736d5037"). InnerVolumeSpecName "kube-api-access-sfpkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:29:59.100159 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.100133 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-token" (OuterVolumeSpecName: "istio-token") pod "a8c70903-ef19-4ef3-894a-ece8736d5037" (UID: "a8c70903-ef19-4ef3-894a-ece8736d5037"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:29:59.157040 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.156995 2573 generic.go:356] "Generic (PLEG): container finished" podID="a8c70903-ef19-4ef3-894a-ece8736d5037" containerID="763aa4e842f3a9aa3d3aa0a18ee99f7b8d7c17bad6799bbb52c80c37f7224d77" exitCode=0 Sep 11 14:29:59.158656 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.157162 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" event={"ID":"a8c70903-ef19-4ef3-894a-ece8736d5037","Type":"ContainerDied","Data":"763aa4e842f3a9aa3d3aa0a18ee99f7b8d7c17bad6799bbb52c80c37f7224d77"} Sep 11 14:29:59.158815 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.158745 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" event={"ID":"a8c70903-ef19-4ef3-894a-ece8736d5037","Type":"ContainerDied","Data":"2c9344eb5dbdae703f304f0eca3fb0fe5d7ec549a603a7fd82d5bdaa72144775"} Sep 11 14:29:59.158889 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.158795 2573 scope.go:117] "RemoveContainer" containerID="763aa4e842f3a9aa3d3aa0a18ee99f7b8d7c17bad6799bbb52c80c37f7224d77" Sep 11 14:29:59.158974 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.157242 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw" Sep 11 14:29:59.171735 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.171709 2573 scope.go:117] "RemoveContainer" containerID="3f70156eead7101a41c0f552d644956b61c5f79cbe58402ac80e9ccec07b055f" Sep 11 14:29:59.182072 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.182056 2573 scope.go:117] "RemoveContainer" containerID="763aa4e842f3a9aa3d3aa0a18ee99f7b8d7c17bad6799bbb52c80c37f7224d77" Sep 11 14:29:59.182368 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:29:59.182343 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"763aa4e842f3a9aa3d3aa0a18ee99f7b8d7c17bad6799bbb52c80c37f7224d77\": container with ID starting with 763aa4e842f3a9aa3d3aa0a18ee99f7b8d7c17bad6799bbb52c80c37f7224d77 not found: ID does not exist" containerID="763aa4e842f3a9aa3d3aa0a18ee99f7b8d7c17bad6799bbb52c80c37f7224d77" Sep 11 14:29:59.182419 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.182380 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"763aa4e842f3a9aa3d3aa0a18ee99f7b8d7c17bad6799bbb52c80c37f7224d77"} err="failed to get container status \"763aa4e842f3a9aa3d3aa0a18ee99f7b8d7c17bad6799bbb52c80c37f7224d77\": rpc error: code = NotFound desc = could not find container \"763aa4e842f3a9aa3d3aa0a18ee99f7b8d7c17bad6799bbb52c80c37f7224d77\": container with ID starting with 763aa4e842f3a9aa3d3aa0a18ee99f7b8d7c17bad6799bbb52c80c37f7224d77 not found: ID does not exist" Sep 11 14:29:59.182419 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.182403 2573 scope.go:117] "RemoveContainer" containerID="3f70156eead7101a41c0f552d644956b61c5f79cbe58402ac80e9ccec07b055f" Sep 11 14:29:59.182665 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:29:59.182646 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f70156eead7101a41c0f552d644956b61c5f79cbe58402ac80e9ccec07b055f\": container with ID starting with 3f70156eead7101a41c0f552d644956b61c5f79cbe58402ac80e9ccec07b055f not found: ID does not exist" containerID="3f70156eead7101a41c0f552d644956b61c5f79cbe58402ac80e9ccec07b055f" Sep 11 14:29:59.182723 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.182674 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f70156eead7101a41c0f552d644956b61c5f79cbe58402ac80e9ccec07b055f"} err="failed to get container status \"3f70156eead7101a41c0f552d644956b61c5f79cbe58402ac80e9ccec07b055f\": rpc error: code = NotFound desc = could not find container \"3f70156eead7101a41c0f552d644956b61c5f79cbe58402ac80e9ccec07b055f\": container with ID starting with 3f70156eead7101a41c0f552d644956b61c5f79cbe58402ac80e9ccec07b055f not found: ID does not exist" Sep 11 14:29:59.188697 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.188673 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw"] Sep 11 14:29:59.194677 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.194646 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b0010444-openshift-default-796b8dcf7b-k6dgw"] Sep 11 14:29:59.199687 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.199667 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:59.199687 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.199688 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:59.199797 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.199698 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfpkm\" (UniqueName: \"kubernetes.io/projected/a8c70903-ef19-4ef3-894a-ece8736d5037-kube-api-access-sfpkm\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:29:59.199797 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:29:59.199708 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c70903-ef19-4ef3-894a-ece8736d5037-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:30:00.158885 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.158849 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818950-4j94k"] Sep 11 14:30:00.160064 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.159997 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c70903-ef19-4ef3-894a-ece8736d5037" containerName="istio-proxy" Sep 11 14:30:00.160064 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.160027 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c70903-ef19-4ef3-894a-ece8736d5037" containerName="istio-proxy" Sep 11 14:30:00.160064 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.160041 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c70903-ef19-4ef3-894a-ece8736d5037" containerName="istio-proxy" Sep 11 14:30:00.160064 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.160050 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c70903-ef19-4ef3-894a-ece8736d5037" containerName="istio-proxy" Sep 11 14:30:00.160358 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.160094 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" containerName="istio-proxy" Sep 11 14:30:00.160358 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.160102 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" containerName="istio-proxy" Sep 11 14:30:00.160358 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.160111 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aacc1bea-98c4-4e8c-912f-f27220b04db1" containerName="istio-proxy" Sep 11 14:30:00.160358 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.160119 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aacc1bea-98c4-4e8c-912f-f27220b04db1" containerName="istio-proxy" Sep 11 14:30:00.160358 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.160132 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" containerName="istio-proxy" Sep 11 14:30:00.160358 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.160140 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" containerName="istio-proxy" Sep 11 14:30:00.160358 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.160253 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c70903-ef19-4ef3-894a-ece8736d5037" containerName="istio-proxy" Sep 11 14:30:00.160358 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.160271 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" containerName="istio-proxy" Sep 11 14:30:00.160358 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.160282 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aacc1bea-98c4-4e8c-912f-f27220b04db1" containerName="istio-proxy" Sep 11 14:30:00.160358 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.160291 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c4d46c5-9e56-4a4c-86d4-d5caeb1a412d" containerName="istio-proxy" Sep 11 14:30:00.165698 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.165669 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-4j94k" Sep 11 14:30:00.166149 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.166128 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39864-29818950-ll4gk"] Sep 11 14:30:00.166860 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.166812 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c70903-ef19-4ef3-894a-ece8736d5037" containerName="istio-proxy" Sep 11 14:30:00.169481 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.169463 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp8gp\"" Sep 11 14:30:00.170072 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.170055 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39864-29818950-ll4gk" Sep 11 14:30:00.176826 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.176664 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818950-4j94k"] Sep 11 14:30:00.181766 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.181748 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-4j94k" Sep 11 14:30:00.191275 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.191238 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8c70903-ef19-4ef3-894a-ece8736d5037" path="/var/lib/kubelet/pods/a8c70903-ef19-4ef3-894a-ece8736d5037/volumes" Sep 11 14:30:00.193188 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.193168 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39864-29818950-ll4gk" Sep 11 14:30:00.201881 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.201851 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39864-29818950-ll4gk"] Sep 11 14:30:00.496484 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.496454 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39864-29818950-ll4gk"] Sep 11 14:30:00.497253 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:30:00.497226 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3f5712_24f6_44bb_8e08_25291dc3f71a.slice/crio-6d6c6d73602496b930e81e00957595bc2e1de0cd23a56714427b645e06c04fc7 WatchSource:0}: Error finding container 6d6c6d73602496b930e81e00957595bc2e1de0cd23a56714427b645e06c04fc7: Status 404 returned error can't find the container with id 6d6c6d73602496b930e81e00957595bc2e1de0cd23a56714427b645e06c04fc7 Sep 11 14:30:00.509387 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:30:00.509338 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788a148c_256b_4d0b_a54b_e42c6d856a2e.slice/crio-66ca5d06cf8c7fd54f6db60709f30ffc640e419f5764f84d305dcafb52cd2e07 WatchSource:0}: Error finding container 66ca5d06cf8c7fd54f6db60709f30ffc640e419f5764f84d305dcafb52cd2e07: Status 404 returned error can't find the container with id 66ca5d06cf8c7fd54f6db60709f30ffc640e419f5764f84d305dcafb52cd2e07 Sep 11 14:30:00.510194 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:00.510169 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818950-4j94k"] Sep 11 14:30:01.177258 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:01.177219 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-4j94k" event={"ID":"788a148c-256b-4d0b-a54b-e42c6d856a2e","Type":"ContainerStarted","Data":"22e430030fb6341bf56d2a918762328ff892b8ebc759e0dd3fd52bada50de47f"} Sep 11 14:30:01.177699 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:01.177287 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-4j94k" event={"ID":"788a148c-256b-4d0b-a54b-e42c6d856a2e","Type":"ContainerStarted","Data":"66ca5d06cf8c7fd54f6db60709f30ffc640e419f5764f84d305dcafb52cd2e07"} Sep 11 14:30:01.178808 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:01.178782 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39864-29818950-ll4gk" event={"ID":"3f3f5712-24f6-44bb-8e08-25291dc3f71a","Type":"ContainerStarted","Data":"c7ccd16a24b817166f89d742a7cbb532f2fdb8e60ec60313ff91001e427e40e2"} Sep 11 14:30:01.178942 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:01.178851 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39864-29818950-ll4gk" event={"ID":"3f3f5712-24f6-44bb-8e08-25291dc3f71a","Type":"ContainerStarted","Data":"6d6c6d73602496b930e81e00957595bc2e1de0cd23a56714427b645e06c04fc7"} Sep 11 14:30:01.200800 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:01.200758 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-4j94k" podStartSLOduration=1.200743553 podStartE2EDuration="1.200743553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:30:01.198343602 +0000 UTC m=+913.535311793" watchObservedRunningTime="2026-09-11 14:30:01.200743553 +0000 UTC m=+913.537711744" Sep 11 14:30:01.221192 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:01.221148 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39864-29818950-ll4gk" podStartSLOduration=1.221136219 podStartE2EDuration="1.221136219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:30:01.219877673 +0000 UTC m=+913.556845865" watchObservedRunningTime="2026-09-11 14:30:01.221136219 +0000 UTC m=+913.558104409" Sep 11 14:30:02.186767 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:02.186729 2573 generic.go:356] "Generic (PLEG): container finished" podID="3f3f5712-24f6-44bb-8e08-25291dc3f71a" containerID="c7ccd16a24b817166f89d742a7cbb532f2fdb8e60ec60313ff91001e427e40e2" exitCode=0 Sep 11 14:30:02.188132 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:02.188106 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39864-29818950-ll4gk" event={"ID":"3f3f5712-24f6-44bb-8e08-25291dc3f71a","Type":"ContainerDied","Data":"c7ccd16a24b817166f89d742a7cbb532f2fdb8e60ec60313ff91001e427e40e2"} Sep 11 14:30:02.188415 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:02.188393 2573 generic.go:356] "Generic (PLEG): container finished" podID="788a148c-256b-4d0b-a54b-e42c6d856a2e" containerID="22e430030fb6341bf56d2a918762328ff892b8ebc759e0dd3fd52bada50de47f" exitCode=0 Sep 11 14:30:02.188492 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:02.188470 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-4j94k" event={"ID":"788a148c-256b-4d0b-a54b-e42c6d856a2e","Type":"ContainerDied","Data":"22e430030fb6341bf56d2a918762328ff892b8ebc759e0dd3fd52bada50de47f"} Sep 11 14:30:03.382619 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:03.382595 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-4j94k" Sep 11 14:30:03.405960 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:03.405937 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39864-29818950-ll4gk" Sep 11 14:30:04.201670 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:04.201646 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39864-29818950-ll4gk" Sep 11 14:30:04.201753 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:04.201662 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39864-29818950-ll4gk" event={"ID":"3f3f5712-24f6-44bb-8e08-25291dc3f71a","Type":"ContainerDied","Data":"6d6c6d73602496b930e81e00957595bc2e1de0cd23a56714427b645e06c04fc7"} Sep 11 14:30:04.201753 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:04.201712 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d6c6d73602496b930e81e00957595bc2e1de0cd23a56714427b645e06c04fc7" Sep 11 14:30:04.203057 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:04.203041 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-4j94k" Sep 11 14:30:04.203057 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:04.203041 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-4j94k" event={"ID":"788a148c-256b-4d0b-a54b-e42c6d856a2e","Type":"ContainerDied","Data":"66ca5d06cf8c7fd54f6db60709f30ffc640e419f5764f84d305dcafb52cd2e07"} Sep 11 14:30:04.203319 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:04.203088 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66ca5d06cf8c7fd54f6db60709f30ffc640e419f5764f84d305dcafb52cd2e07" Sep 11 14:30:08.230443 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:08.230352 2573 generic.go:356] "Generic (PLEG): container finished" podID="630cd4b9-d48c-4605-be3c-b2f1eedc70ef" containerID="94eb2a29465bc4fea502c4d41acbbd63b6b5d2aca85868194588086e5462ecd2" exitCode=0 Sep 11 14:30:08.230443 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:08.230406 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" event={"ID":"630cd4b9-d48c-4605-be3c-b2f1eedc70ef","Type":"ContainerDied","Data":"94eb2a29465bc4fea502c4d41acbbd63b6b5d2aca85868194588086e5462ecd2"} Sep 11 14:30:08.230914 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:08.230769 2573 scope.go:117] "RemoveContainer" containerID="94eb2a29465bc4fea502c4d41acbbd63b6b5d2aca85868194588086e5462ecd2" Sep 11 14:30:08.231812 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:08.231785 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:30:08.231908 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:08.231864 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:30:08.231908 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:08.231897 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:30:09.242962 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:09.242922 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" event={"ID":"630cd4b9-d48c-4605-be3c-b2f1eedc70ef","Type":"ContainerStarted","Data":"67936a31b812ed351ad462fb07c9344abf845daac2696bc8514abe346c239967"} Sep 11 14:30:09.415907 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:09.415873 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:30:09.416089 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:09.416068 2573 patch_prober.go:28] interesting pod/e2e-explicit-696fee-openshift-default-848b84776-vwd7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:30:09.416142 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:09.416126 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" podUID="630cd4b9-d48c-4605-be3c-b2f1eedc70ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 14:30:10.416214 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:10.416176 2573 patch_prober.go:28] interesting pod/e2e-explicit-696fee-openshift-default-848b84776-vwd7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:30:10.416696 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:10.416239 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" podUID="630cd4b9-d48c-4605-be3c-b2f1eedc70ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 14:30:11.415703 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:11.415667 2573 patch_prober.go:28] interesting pod/e2e-explicit-696fee-openshift-default-848b84776-vwd7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:30:11.415907 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:11.415732 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" podUID="630cd4b9-d48c-4605-be3c-b2f1eedc70ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 14:30:12.415470 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:12.415431 2573 patch_prober.go:28] interesting pod/e2e-explicit-696fee-openshift-default-848b84776-vwd7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:30:12.415890 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:12.415487 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" podUID="630cd4b9-d48c-4605-be3c-b2f1eedc70ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 14:30:13.415493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:13.415457 2573 patch_prober.go:28] interesting pod/e2e-explicit-696fee-openshift-default-848b84776-vwd7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:30:13.415913 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:13.415537 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" podUID="630cd4b9-d48c-4605-be3c-b2f1eedc70ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 14:30:14.419827 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:14.419795 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:30:14.420202 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:14.420143 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:30:14.420810 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:14.420794 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:30:58.053644 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:58.053610 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t"] Sep 11 14:30:58.054125 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:58.053890 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" podUID="630cd4b9-d48c-4605-be3c-b2f1eedc70ef" containerName="istio-proxy" containerID="cri-o://67936a31b812ed351ad462fb07c9344abf845daac2696bc8514abe346c239967" gracePeriod=30 Sep 11 14:30:58.928379 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:58.928341 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2"] Sep 11 14:30:58.928780 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:58.928767 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3f5712-24f6-44bb-8e08-25291dc3f71a" containerName="cleanup" Sep 11 14:30:58.928860 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:58.928781 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3f5712-24f6-44bb-8e08-25291dc3f71a" containerName="cleanup" Sep 11 14:30:58.928860 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:58.928802 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="788a148c-256b-4d0b-a54b-e42c6d856a2e" containerName="cleanup" Sep 11 14:30:58.928860 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:58.928807 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="788a148c-256b-4d0b-a54b-e42c6d856a2e" containerName="cleanup" Sep 11 14:30:58.929021 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:58.928887 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3f5712-24f6-44bb-8e08-25291dc3f71a" containerName="cleanup" Sep 11 14:30:58.929021 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:58.928895 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="788a148c-256b-4d0b-a54b-e42c6d856a2e" containerName="cleanup" Sep 11 14:30:58.933285 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:58.933264 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:58.938395 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:58.938376 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-e1e477-openshift-default-dockercfg-g9s4p\"" Sep 11 14:30:59.003690 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.003661 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78b29\" (UniqueName: \"kubernetes.io/projected/3d84dc43-8e9b-426c-a2de-53bf78da3f62-kube-api-access-78b29\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.003826 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.003748 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-workload-socket\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.003826 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.003811 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-podinfo\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.003975 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.003865 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-credential-socket\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.003975 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.003892 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istiod-ca-cert\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.003975 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.003935 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-token\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.004170 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.004145 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-envoy\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.004263 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.004227 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-data\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.004323 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.004265 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-workload-certs\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.018818 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.018785 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2"] Sep 11 14:30:59.105061 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.105030 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78b29\" (UniqueName: \"kubernetes.io/projected/3d84dc43-8e9b-426c-a2de-53bf78da3f62-kube-api-access-78b29\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.105529 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.105065 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-workload-socket\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.105529 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.105090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-podinfo\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.105529 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.105106 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-credential-socket\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.105529 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.105130 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istiod-ca-cert\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.105529 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.105253 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-token\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.105529 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.105318 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-envoy\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.105529 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.105373 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-data\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.105529 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.105410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-workload-certs\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.105992 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.105541 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-workload-socket\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.105992 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.105589 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-credential-socket\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.105992 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.105759 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-workload-certs\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.105992 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.105785 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-data\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.106190 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.106110 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istiod-ca-cert\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.107606 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.107585 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-envoy\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.107703 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.107678 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-podinfo\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.115320 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.115299 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78b29\" (UniqueName: \"kubernetes.io/projected/3d84dc43-8e9b-426c-a2de-53bf78da3f62-kube-api-access-78b29\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.115441 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.115387 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-token\") pod \"e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.244531 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.244450 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:30:59.422996 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.422959 2573 patch_prober.go:28] interesting pod/e2e-explicit-696fee-openshift-default-848b84776-vwd7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:30:59.423155 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.423022 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" podUID="630cd4b9-d48c-4605-be3c-b2f1eedc70ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:30:59.536313 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.536286 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2"] Sep 11 14:30:59.536609 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:30:59.536576 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d84dc43_8e9b_426c_a2de_53bf78da3f62.slice/crio-9e99bc862f308611014864186ad0fc098c963f47778afcc03a7b388a01d1709f WatchSource:0}: Error finding container 9e99bc862f308611014864186ad0fc098c963f47778afcc03a7b388a01d1709f: Status 404 returned error can't find the container with id 9e99bc862f308611014864186ad0fc098c963f47778afcc03a7b388a01d1709f Sep 11 14:30:59.539091 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.539044 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:30:59.539192 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.539136 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:30:59.539192 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.539176 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:30:59.551584 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:30:59.551555 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" event={"ID":"3d84dc43-8e9b-426c-a2de-53bf78da3f62","Type":"ContainerStarted","Data":"9e99bc862f308611014864186ad0fc098c963f47778afcc03a7b388a01d1709f"} Sep 11 14:31:00.560515 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:00.560478 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" event={"ID":"3d84dc43-8e9b-426c-a2de-53bf78da3f62","Type":"ContainerStarted","Data":"8d1788f916d04aaec12cbf8206e2fd5be1de8a65e77d4c7fd40461680d9c7444"} Sep 11 14:31:00.584222 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:00.584170 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" podStartSLOduration=2.584155977 podStartE2EDuration="2.584155977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:00.582598693 +0000 UTC m=+972.919566884" watchObservedRunningTime="2026-09-11 14:31:00.584155977 +0000 UTC m=+972.921124241" Sep 11 14:31:01.244925 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:01.244889 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:31:01.249887 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:01.249861 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:31:01.567554 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:01.567470 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:31:01.568346 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:01.568327 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:31:03.301772 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.301749 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:31:03.351979 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.351954 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-credential-socket\") pod \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " Sep 11 14:31:03.352154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.351999 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-workload-socket\") pod \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " Sep 11 14:31:03.352154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.352035 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-podinfo\") pod \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " Sep 11 14:31:03.352154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.352053 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-workload-certs\") pod \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " Sep 11 14:31:03.352154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.352069 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-token\") pod \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " Sep 11 14:31:03.352154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.352112 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istiod-ca-cert\") pod \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " Sep 11 14:31:03.352154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.352141 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-data\") pod \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " Sep 11 14:31:03.352518 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.352183 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-497zm\" (UniqueName: \"kubernetes.io/projected/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-kube-api-access-497zm\") pod \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " Sep 11 14:31:03.352518 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.352223 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-envoy\") pod \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\" (UID: \"630cd4b9-d48c-4605-be3c-b2f1eedc70ef\") " Sep 11 14:31:03.352518 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.352265 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "630cd4b9-d48c-4605-be3c-b2f1eedc70ef" (UID: "630cd4b9-d48c-4605-be3c-b2f1eedc70ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:03.352518 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.352279 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "630cd4b9-d48c-4605-be3c-b2f1eedc70ef" (UID: "630cd4b9-d48c-4605-be3c-b2f1eedc70ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:03.352518 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.352478 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "630cd4b9-d48c-4605-be3c-b2f1eedc70ef" (UID: "630cd4b9-d48c-4605-be3c-b2f1eedc70ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:03.352518 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.352483 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "630cd4b9-d48c-4605-be3c-b2f1eedc70ef" (UID: "630cd4b9-d48c-4605-be3c-b2f1eedc70ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:03.352853 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.352733 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:03.352853 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.352755 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:03.352853 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.352771 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:03.352853 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.352785 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:03.352853 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.352806 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "630cd4b9-d48c-4605-be3c-b2f1eedc70ef" (UID: "630cd4b9-d48c-4605-be3c-b2f1eedc70ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:31:03.354509 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.354477 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "630cd4b9-d48c-4605-be3c-b2f1eedc70ef" (UID: "630cd4b9-d48c-4605-be3c-b2f1eedc70ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:31:03.354709 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.354688 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "630cd4b9-d48c-4605-be3c-b2f1eedc70ef" (UID: "630cd4b9-d48c-4605-be3c-b2f1eedc70ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:31:03.354775 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.354692 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-kube-api-access-497zm" (OuterVolumeSpecName: "kube-api-access-497zm") pod "630cd4b9-d48c-4605-be3c-b2f1eedc70ef" (UID: "630cd4b9-d48c-4605-be3c-b2f1eedc70ef"). InnerVolumeSpecName "kube-api-access-497zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:31:03.354775 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.354718 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "630cd4b9-d48c-4605-be3c-b2f1eedc70ef" (UID: "630cd4b9-d48c-4605-be3c-b2f1eedc70ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:03.454377 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.454343 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:03.454377 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.454388 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:03.454585 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.454404 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:03.454585 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.454416 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-497zm\" (UniqueName: \"kubernetes.io/projected/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-kube-api-access-497zm\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:03.454585 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.454428 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630cd4b9-d48c-4605-be3c-b2f1eedc70ef-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:03.580509 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.580476 2573 generic.go:356] "Generic (PLEG): container finished" podID="630cd4b9-d48c-4605-be3c-b2f1eedc70ef" containerID="67936a31b812ed351ad462fb07c9344abf845daac2696bc8514abe346c239967" exitCode=0 Sep 11 14:31:03.580671 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.580527 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" Sep 11 14:31:03.580671 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.580562 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" event={"ID":"630cd4b9-d48c-4605-be3c-b2f1eedc70ef","Type":"ContainerDied","Data":"67936a31b812ed351ad462fb07c9344abf845daac2696bc8514abe346c239967"} Sep 11 14:31:03.580671 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.580619 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t" event={"ID":"630cd4b9-d48c-4605-be3c-b2f1eedc70ef","Type":"ContainerDied","Data":"7e86c808a6aeacbed6c83d32c2eed9fb954a32f74bb63ed9c424bb6c061145ec"} Sep 11 14:31:03.580671 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.580652 2573 scope.go:117] "RemoveContainer" containerID="67936a31b812ed351ad462fb07c9344abf845daac2696bc8514abe346c239967" Sep 11 14:31:03.595163 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.595142 2573 scope.go:117] "RemoveContainer" containerID="94eb2a29465bc4fea502c4d41acbbd63b6b5d2aca85868194588086e5462ecd2" Sep 11 14:31:03.604013 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.603995 2573 scope.go:117] "RemoveContainer" containerID="67936a31b812ed351ad462fb07c9344abf845daac2696bc8514abe346c239967" Sep 11 14:31:03.604265 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:31:03.604248 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67936a31b812ed351ad462fb07c9344abf845daac2696bc8514abe346c239967\": container with ID starting with 67936a31b812ed351ad462fb07c9344abf845daac2696bc8514abe346c239967 not found: ID does not exist" containerID="67936a31b812ed351ad462fb07c9344abf845daac2696bc8514abe346c239967" Sep 11 14:31:03.604306 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.604274 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67936a31b812ed351ad462fb07c9344abf845daac2696bc8514abe346c239967"} err="failed to get container status \"67936a31b812ed351ad462fb07c9344abf845daac2696bc8514abe346c239967\": rpc error: code = NotFound desc = could not find container \"67936a31b812ed351ad462fb07c9344abf845daac2696bc8514abe346c239967\": container with ID starting with 67936a31b812ed351ad462fb07c9344abf845daac2696bc8514abe346c239967 not found: ID does not exist" Sep 11 14:31:03.604306 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.604289 2573 scope.go:117] "RemoveContainer" containerID="94eb2a29465bc4fea502c4d41acbbd63b6b5d2aca85868194588086e5462ecd2" Sep 11 14:31:03.604814 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:31:03.604554 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94eb2a29465bc4fea502c4d41acbbd63b6b5d2aca85868194588086e5462ecd2\": container with ID starting with 94eb2a29465bc4fea502c4d41acbbd63b6b5d2aca85868194588086e5462ecd2 not found: ID does not exist" containerID="94eb2a29465bc4fea502c4d41acbbd63b6b5d2aca85868194588086e5462ecd2" Sep 11 14:31:03.604814 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.604603 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94eb2a29465bc4fea502c4d41acbbd63b6b5d2aca85868194588086e5462ecd2"} err="failed to get container status \"94eb2a29465bc4fea502c4d41acbbd63b6b5d2aca85868194588086e5462ecd2\": rpc error: code = NotFound desc = could not find container \"94eb2a29465bc4fea502c4d41acbbd63b6b5d2aca85868194588086e5462ecd2\": container with ID starting with 94eb2a29465bc4fea502c4d41acbbd63b6b5d2aca85868194588086e5462ecd2 not found: ID does not exist" Sep 11 14:31:03.608367 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.608344 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t"] Sep 11 14:31:03.614953 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:03.614929 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-696fee-openshift-default-848b84776-vwd7t"] Sep 11 14:31:04.188864 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:04.188813 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="630cd4b9-d48c-4605-be3c-b2f1eedc70ef" path="/var/lib/kubelet/pods/630cd4b9-d48c-4605-be3c-b2f1eedc70ef/volumes" Sep 11 14:31:11.774696 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:11.774615 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc"] Sep 11 14:31:26.793679 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:26.793586 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" podUID="12f35580-a375-449d-a57b-3bee55c487dd" containerName="main" containerID="cri-o://56cb37237b79b903cd3e466e377b4bcf433732e1b543c6fb2611f5d3d433ea57" gracePeriod=45 Sep 11 14:31:27.049474 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.049416 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:31:27.075177 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.075150 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zbfp\" (UniqueName: \"kubernetes.io/projected/12f35580-a375-449d-a57b-3bee55c487dd-kube-api-access-9zbfp\") pod \"12f35580-a375-449d-a57b-3bee55c487dd\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " Sep 11 14:31:27.075313 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.075211 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-model-cache\") pod \"12f35580-a375-449d-a57b-3bee55c487dd\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " Sep 11 14:31:27.075313 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.075239 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f35580-a375-449d-a57b-3bee55c487dd-tls-certs\") pod \"12f35580-a375-449d-a57b-3bee55c487dd\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " Sep 11 14:31:27.075313 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.075254 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-dshm\") pod \"12f35580-a375-449d-a57b-3bee55c487dd\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " Sep 11 14:31:27.075454 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.075314 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-tmp-dir\") pod \"12f35580-a375-449d-a57b-3bee55c487dd\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " Sep 11 14:31:27.075454 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.075346 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-home\") pod \"12f35580-a375-449d-a57b-3bee55c487dd\" (UID: \"12f35580-a375-449d-a57b-3bee55c487dd\") " Sep 11 14:31:27.075555 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.075536 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "12f35580-a375-449d-a57b-3bee55c487dd" (UID: "12f35580-a375-449d-a57b-3bee55c487dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:27.075655 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.075624 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12f35580-a375-449d-a57b-3bee55c487dd" (UID: "12f35580-a375-449d-a57b-3bee55c487dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:27.075724 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.075644 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-home" (OuterVolumeSpecName: "home") pod "12f35580-a375-449d-a57b-3bee55c487dd" (UID: "12f35580-a375-449d-a57b-3bee55c487dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:27.075765 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.075725 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-model-cache\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:27.075765 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.075741 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-tmp-dir\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:27.075765 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.075749 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-home\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:27.077459 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.077437 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12f35580-a375-449d-a57b-3bee55c487dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12f35580-a375-449d-a57b-3bee55c487dd" (UID: "12f35580-a375-449d-a57b-3bee55c487dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:31:27.077742 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.077725 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-dshm" (OuterVolumeSpecName: "dshm") pod "12f35580-a375-449d-a57b-3bee55c487dd" (UID: "12f35580-a375-449d-a57b-3bee55c487dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:27.077742 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.077733 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f35580-a375-449d-a57b-3bee55c487dd-kube-api-access-9zbfp" (OuterVolumeSpecName: "kube-api-access-9zbfp") pod "12f35580-a375-449d-a57b-3bee55c487dd" (UID: "12f35580-a375-449d-a57b-3bee55c487dd"). InnerVolumeSpecName "kube-api-access-9zbfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:31:27.176992 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.176960 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zbfp\" (UniqueName: \"kubernetes.io/projected/12f35580-a375-449d-a57b-3bee55c487dd-kube-api-access-9zbfp\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:27.176992 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.176986 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f35580-a375-449d-a57b-3bee55c487dd-tls-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:27.176992 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.176996 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f35580-a375-449d-a57b-3bee55c487dd-dshm\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:27.350262 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.350217 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2"] Sep 11 14:31:27.350467 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.350441 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" podUID="3d84dc43-8e9b-426c-a2de-53bf78da3f62" containerName="istio-proxy" containerID="cri-o://8d1788f916d04aaec12cbf8206e2fd5be1de8a65e77d4c7fd40461680d9c7444" gracePeriod=30 Sep 11 14:31:27.725582 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.725492 2573 generic.go:356] "Generic (PLEG): container finished" podID="12f35580-a375-449d-a57b-3bee55c487dd" containerID="56cb37237b79b903cd3e466e377b4bcf433732e1b543c6fb2611f5d3d433ea57" exitCode=0 Sep 11 14:31:27.725582 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.725549 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" Sep 11 14:31:27.725810 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.725604 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" event={"ID":"12f35580-a375-449d-a57b-3bee55c487dd","Type":"ContainerDied","Data":"56cb37237b79b903cd3e466e377b4bcf433732e1b543c6fb2611f5d3d433ea57"} Sep 11 14:31:27.725810 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.725660 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc" event={"ID":"12f35580-a375-449d-a57b-3bee55c487dd","Type":"ContainerDied","Data":"6906459a0f5cfa6d799cfe3e44fc6a862b7b0966e9032401ed895c64509d019b"} Sep 11 14:31:27.725810 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.725684 2573 scope.go:117] "RemoveContainer" containerID="56cb37237b79b903cd3e466e377b4bcf433732e1b543c6fb2611f5d3d433ea57" Sep 11 14:31:27.735607 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.735593 2573 scope.go:117] "RemoveContainer" containerID="56cb37237b79b903cd3e466e377b4bcf433732e1b543c6fb2611f5d3d433ea57" Sep 11 14:31:27.735885 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:31:27.735865 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56cb37237b79b903cd3e466e377b4bcf433732e1b543c6fb2611f5d3d433ea57\": container with ID starting with 56cb37237b79b903cd3e466e377b4bcf433732e1b543c6fb2611f5d3d433ea57 not found: ID does not exist" containerID="56cb37237b79b903cd3e466e377b4bcf433732e1b543c6fb2611f5d3d433ea57" Sep 11 14:31:27.735968 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.735891 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56cb37237b79b903cd3e466e377b4bcf433732e1b543c6fb2611f5d3d433ea57"} err="failed to get container status \"56cb37237b79b903cd3e466e377b4bcf433732e1b543c6fb2611f5d3d433ea57\": rpc error: code = NotFound desc = could not find container \"56cb37237b79b903cd3e466e377b4bcf433732e1b543c6fb2611f5d3d433ea57\": container with ID starting with 56cb37237b79b903cd3e466e377b4bcf433732e1b543c6fb2611f5d3d433ea57 not found: ID does not exist" Sep 11 14:31:27.748451 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.748430 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc"] Sep 11 14:31:27.752483 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:27.752460 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a39864/test-model-a39864-kserve-655b77cb9f-kqqxc"] Sep 11 14:31:28.187201 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:28.187165 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12f35580-a375-449d-a57b-3bee55c487dd" path="/var/lib/kubelet/pods/12f35580-a375-449d-a57b-3bee55c487dd/volumes" Sep 11 14:31:31.569717 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:31.569675 2573 patch_prober.go:28] interesting pod/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:31:31.570110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:31.569744 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" podUID="3d84dc43-8e9b-426c-a2de-53bf78da3f62" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:31:32.603635 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.603611 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:31:32.632082 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632020 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78b29\" (UniqueName: \"kubernetes.io/projected/3d84dc43-8e9b-426c-a2de-53bf78da3f62-kube-api-access-78b29\") pod \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " Sep 11 14:31:32.632082 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632053 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-workload-socket\") pod \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " Sep 11 14:31:32.632082 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632076 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-credential-socket\") pod \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " Sep 11 14:31:32.632317 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632109 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-workload-certs\") pod \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " Sep 11 14:31:32.632317 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632131 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-token\") pod \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " Sep 11 14:31:32.632317 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632189 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-podinfo\") pod \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " Sep 11 14:31:32.632317 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632214 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-envoy\") pod \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " Sep 11 14:31:32.632317 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632281 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-data\") pod \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " Sep 11 14:31:32.632317 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632313 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istiod-ca-cert\") pod \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\" (UID: \"3d84dc43-8e9b-426c-a2de-53bf78da3f62\") " Sep 11 14:31:32.632600 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632337 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d84dc43-8e9b-426c-a2de-53bf78da3f62" (UID: "3d84dc43-8e9b-426c-a2de-53bf78da3f62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:32.632600 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632351 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d84dc43-8e9b-426c-a2de-53bf78da3f62" (UID: "3d84dc43-8e9b-426c-a2de-53bf78da3f62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:32.632600 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632367 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d84dc43-8e9b-426c-a2de-53bf78da3f62" (UID: "3d84dc43-8e9b-426c-a2de-53bf78da3f62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:32.632600 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632578 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d84dc43-8e9b-426c-a2de-53bf78da3f62" (UID: "3d84dc43-8e9b-426c-a2de-53bf78da3f62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:32.632806 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632706 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:32.632806 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632724 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:32.632806 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632739 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:32.632806 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632753 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:32.633006 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.632852 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d84dc43-8e9b-426c-a2de-53bf78da3f62" (UID: "3d84dc43-8e9b-426c-a2de-53bf78da3f62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:31:32.634629 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.634608 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d84dc43-8e9b-426c-a2de-53bf78da3f62-kube-api-access-78b29" (OuterVolumeSpecName: "kube-api-access-78b29") pod "3d84dc43-8e9b-426c-a2de-53bf78da3f62" (UID: "3d84dc43-8e9b-426c-a2de-53bf78da3f62"). InnerVolumeSpecName "kube-api-access-78b29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:31:32.634741 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.634706 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d84dc43-8e9b-426c-a2de-53bf78da3f62" (UID: "3d84dc43-8e9b-426c-a2de-53bf78da3f62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:31:32.634741 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.634722 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d84dc43-8e9b-426c-a2de-53bf78da3f62" (UID: "3d84dc43-8e9b-426c-a2de-53bf78da3f62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:32.634987 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.634783 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d84dc43-8e9b-426c-a2de-53bf78da3f62" (UID: "3d84dc43-8e9b-426c-a2de-53bf78da3f62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:31:32.734296 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.734266 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:32.734296 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.734289 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:32.734296 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.734298 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:32.734501 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.734306 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78b29\" (UniqueName: \"kubernetes.io/projected/3d84dc43-8e9b-426c-a2de-53bf78da3f62-kube-api-access-78b29\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:32.734501 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.734316 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d84dc43-8e9b-426c-a2de-53bf78da3f62-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:32.762947 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.762914 2573 generic.go:356] "Generic (PLEG): container finished" podID="3d84dc43-8e9b-426c-a2de-53bf78da3f62" containerID="8d1788f916d04aaec12cbf8206e2fd5be1de8a65e77d4c7fd40461680d9c7444" exitCode=0 Sep 11 14:31:32.763061 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.762966 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" Sep 11 14:31:32.763061 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.763000 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" event={"ID":"3d84dc43-8e9b-426c-a2de-53bf78da3f62","Type":"ContainerDied","Data":"8d1788f916d04aaec12cbf8206e2fd5be1de8a65e77d4c7fd40461680d9c7444"} Sep 11 14:31:32.763061 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.763051 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2" event={"ID":"3d84dc43-8e9b-426c-a2de-53bf78da3f62","Type":"ContainerDied","Data":"9e99bc862f308611014864186ad0fc098c963f47778afcc03a7b388a01d1709f"} Sep 11 14:31:32.763167 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.763073 2573 scope.go:117] "RemoveContainer" containerID="8d1788f916d04aaec12cbf8206e2fd5be1de8a65e77d4c7fd40461680d9c7444" Sep 11 14:31:32.773476 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.773459 2573 scope.go:117] "RemoveContainer" containerID="8d1788f916d04aaec12cbf8206e2fd5be1de8a65e77d4c7fd40461680d9c7444" Sep 11 14:31:32.773729 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:31:32.773711 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d1788f916d04aaec12cbf8206e2fd5be1de8a65e77d4c7fd40461680d9c7444\": container with ID starting with 8d1788f916d04aaec12cbf8206e2fd5be1de8a65e77d4c7fd40461680d9c7444 not found: ID does not exist" containerID="8d1788f916d04aaec12cbf8206e2fd5be1de8a65e77d4c7fd40461680d9c7444" Sep 11 14:31:32.773787 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.773744 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d1788f916d04aaec12cbf8206e2fd5be1de8a65e77d4c7fd40461680d9c7444"} err="failed to get container status \"8d1788f916d04aaec12cbf8206e2fd5be1de8a65e77d4c7fd40461680d9c7444\": rpc error: code = NotFound desc = could not find container \"8d1788f916d04aaec12cbf8206e2fd5be1de8a65e77d4c7fd40461680d9c7444\": container with ID starting with 8d1788f916d04aaec12cbf8206e2fd5be1de8a65e77d4c7fd40461680d9c7444 not found: ID does not exist" Sep 11 14:31:32.787392 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.787370 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2"] Sep 11 14:31:32.792657 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:32.792635 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-e1e477-openshift-default-5c7dc9b574-m29g2"] Sep 11 14:31:34.186980 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:34.186947 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d84dc43-8e9b-426c-a2de-53bf78da3f62" path="/var/lib/kubelet/pods/3d84dc43-8e9b-426c-a2de-53bf78da3f62/volumes" Sep 11 14:31:35.364055 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.364024 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv"] Sep 11 14:31:35.364430 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.364417 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="630cd4b9-d48c-4605-be3c-b2f1eedc70ef" containerName="istio-proxy" Sep 11 14:31:35.364430 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.364430 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="630cd4b9-d48c-4605-be3c-b2f1eedc70ef" containerName="istio-proxy" Sep 11 14:31:35.364509 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.364446 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12f35580-a375-449d-a57b-3bee55c487dd" containerName="main" Sep 11 14:31:35.364509 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.364453 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="12f35580-a375-449d-a57b-3bee55c487dd" containerName="main" Sep 11 14:31:35.364509 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.364459 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="630cd4b9-d48c-4605-be3c-b2f1eedc70ef" containerName="istio-proxy" Sep 11 14:31:35.364509 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.364466 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="630cd4b9-d48c-4605-be3c-b2f1eedc70ef" containerName="istio-proxy" Sep 11 14:31:35.364509 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.364484 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d84dc43-8e9b-426c-a2de-53bf78da3f62" containerName="istio-proxy" Sep 11 14:31:35.364509 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.364489 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d84dc43-8e9b-426c-a2de-53bf78da3f62" containerName="istio-proxy" Sep 11 14:31:35.364690 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.364548 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="12f35580-a375-449d-a57b-3bee55c487dd" containerName="main" Sep 11 14:31:35.364690 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.364557 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="630cd4b9-d48c-4605-be3c-b2f1eedc70ef" containerName="istio-proxy" Sep 11 14:31:35.364690 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.364563 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d84dc43-8e9b-426c-a2de-53bf78da3f62" containerName="istio-proxy" Sep 11 14:31:35.364690 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.364573 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="630cd4b9-d48c-4605-be3c-b2f1eedc70ef" containerName="istio-proxy" Sep 11 14:31:35.369116 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.369099 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.372532 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.372511 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-02723795-openshift-default-dockercfg-t6rhm\"" Sep 11 14:31:35.380985 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.380963 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv"] Sep 11 14:31:35.460636 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.460585 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa1b2aea-b48f-4197-8386-c6072ad36889-istiod-ca-cert\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.460863 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.460716 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-envoy\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.460863 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.460809 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-workload-certs\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.461001 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.460893 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-data\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.461001 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.460995 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-workload-socket\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.461126 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.461082 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkww7\" (UniqueName: \"kubernetes.io/projected/fa1b2aea-b48f-4197-8386-c6072ad36889-kube-api-access-vkww7\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.461367 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.461332 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-credential-socket\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.461446 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.461398 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-token\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.461446 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.461428 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-podinfo\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.562942 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.562907 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-token\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.562942 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.562945 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-podinfo\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.563208 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.562987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa1b2aea-b48f-4197-8386-c6072ad36889-istiod-ca-cert\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.563208 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.563020 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-envoy\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.563208 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.563048 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-workload-certs\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.563208 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.563073 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-data\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.563208 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.563124 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-workload-socket\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.563208 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.563170 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkww7\" (UniqueName: \"kubernetes.io/projected/fa1b2aea-b48f-4197-8386-c6072ad36889-kube-api-access-vkww7\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.563492 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.563223 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-credential-socket\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.563646 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.563615 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-workload-socket\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.563646 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.563633 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-data\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.563812 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.563654 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-credential-socket\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.563876 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.563861 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-workload-certs\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.564015 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.563996 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa1b2aea-b48f-4197-8386-c6072ad36889-istiod-ca-cert\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.565919 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.565889 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-envoy\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.566067 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.566050 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-podinfo\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.572657 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.572629 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkww7\" (UniqueName: \"kubernetes.io/projected/fa1b2aea-b48f-4197-8386-c6072ad36889-kube-api-access-vkww7\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.572886 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.572850 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-token\") pod \"e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.680547 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.680464 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:35.970631 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.970599 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv"] Sep 11 14:31:35.971679 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:31:35.971646 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1b2aea_b48f_4197_8386_c6072ad36889.slice/crio-a6aafdf92076da8ceab8ac0ed1b7a6d3766ca420f748241f6296ee777c656ca4 WatchSource:0}: Error finding container a6aafdf92076da8ceab8ac0ed1b7a6d3766ca420f748241f6296ee777c656ca4: Status 404 returned error can't find the container with id a6aafdf92076da8ceab8ac0ed1b7a6d3766ca420f748241f6296ee777c656ca4 Sep 11 14:31:35.974079 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.974040 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:31:35.974173 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.974137 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:31:35.974214 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:35.974181 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:31:36.789894 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:36.789853 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" event={"ID":"fa1b2aea-b48f-4197-8386-c6072ad36889","Type":"ContainerStarted","Data":"77deee2cb5bc3f44a6fd837a3ce91b543fb83dd45605523d8a891da2d3e19d04"} Sep 11 14:31:36.790290 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:36.789920 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" event={"ID":"fa1b2aea-b48f-4197-8386-c6072ad36889","Type":"ContainerStarted","Data":"a6aafdf92076da8ceab8ac0ed1b7a6d3766ca420f748241f6296ee777c656ca4"} Sep 11 14:31:36.819299 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:36.819252 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" podStartSLOduration=1.819238319 podStartE2EDuration="1.819238319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:36.816058463 +0000 UTC m=+1009.153026664" watchObservedRunningTime="2026-09-11 14:31:36.819238319 +0000 UTC m=+1009.156206510" Sep 11 14:31:37.100297 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:37.100260 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39864-29818950-ll4gk"] Sep 11 14:31:37.103488 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:37.103461 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a39864-29818950-ll4gk"] Sep 11 14:31:37.680815 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:37.680782 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:37.686092 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:37.686066 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:37.796648 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:37.796624 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:37.797363 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:37.797345 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:38.189439 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:38.189404 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f3f5712-24f6-44bb-8e08-25291dc3f71a" path="/var/lib/kubelet/pods/3f3f5712-24f6-44bb-8e08-25291dc3f71a/volumes" Sep 11 14:31:44.628460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:44.628416 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd"] Sep 11 14:31:44.628851 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:44.628697 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" podUID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerName="istio-proxy" containerID="cri-o://e5568f1fe185f6c35721d760934d2296a933007c57574c71eab154d17a72103a" gracePeriod=30 Sep 11 14:31:47.365667 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:47.365598 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:31:47.366046 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:47.365700 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" podUID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:31:49.875539 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:49.875503 2573 generic.go:356] "Generic (PLEG): container finished" podID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerID="e5568f1fe185f6c35721d760934d2296a933007c57574c71eab154d17a72103a" exitCode=0 Sep 11 14:31:49.875936 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:49.875550 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" event={"ID":"08c123b1-afa2-4560-9bf1-bb056db36f81","Type":"ContainerDied","Data":"e5568f1fe185f6c35721d760934d2296a933007c57574c71eab154d17a72103a"} Sep 11 14:31:49.875936 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:49.875606 2573 scope.go:117] "RemoveContainer" containerID="1b642ba6f8f54dd4f773f9dbb2b2f32bba02335ea1d3b6434953596155d0a949" Sep 11 14:31:49.911252 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:49.911198 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:31:50.004463 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.004429 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-workload-certs\") pod \"08c123b1-afa2-4560-9bf1-bb056db36f81\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " Sep 11 14:31:50.004663 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.004477 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcc64\" (UniqueName: \"kubernetes.io/projected/08c123b1-afa2-4560-9bf1-bb056db36f81-kube-api-access-xcc64\") pod \"08c123b1-afa2-4560-9bf1-bb056db36f81\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " Sep 11 14:31:50.004663 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.004506 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-envoy\") pod \"08c123b1-afa2-4560-9bf1-bb056db36f81\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " Sep 11 14:31:50.004663 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.004574 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c123b1-afa2-4560-9bf1-bb056db36f81-istiod-ca-cert\") pod \"08c123b1-afa2-4560-9bf1-bb056db36f81\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " Sep 11 14:31:50.004663 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.004607 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-data\") pod \"08c123b1-afa2-4560-9bf1-bb056db36f81\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " Sep 11 14:31:50.004663 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.004638 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-token\") pod \"08c123b1-afa2-4560-9bf1-bb056db36f81\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " Sep 11 14:31:50.004965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.004671 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08c123b1-afa2-4560-9bf1-bb056db36f81" (UID: "08c123b1-afa2-4560-9bf1-bb056db36f81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:50.004965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.004715 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-workload-socket\") pod \"08c123b1-afa2-4560-9bf1-bb056db36f81\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " Sep 11 14:31:50.004965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.004779 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-credential-socket\") pod \"08c123b1-afa2-4560-9bf1-bb056db36f81\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " Sep 11 14:31:50.004965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.004800 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-podinfo\") pod \"08c123b1-afa2-4560-9bf1-bb056db36f81\" (UID: \"08c123b1-afa2-4560-9bf1-bb056db36f81\") " Sep 11 14:31:50.004965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.004870 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-data" (OuterVolumeSpecName: "istio-data") pod "08c123b1-afa2-4560-9bf1-bb056db36f81" (UID: "08c123b1-afa2-4560-9bf1-bb056db36f81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:50.004965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.004943 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c123b1-afa2-4560-9bf1-bb056db36f81-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08c123b1-afa2-4560-9bf1-bb056db36f81" (UID: "08c123b1-afa2-4560-9bf1-bb056db36f81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:31:50.005248 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.005023 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08c123b1-afa2-4560-9bf1-bb056db36f81" (UID: "08c123b1-afa2-4560-9bf1-bb056db36f81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:50.005248 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.005059 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08c123b1-afa2-4560-9bf1-bb056db36f81" (UID: "08c123b1-afa2-4560-9bf1-bb056db36f81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:50.005248 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.005206 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:50.005248 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.005230 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c123b1-afa2-4560-9bf1-bb056db36f81-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:50.005248 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.005242 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:50.005460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.005254 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:50.005460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.005301 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:50.006949 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.006913 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08c123b1-afa2-4560-9bf1-bb056db36f81" (UID: "08c123b1-afa2-4560-9bf1-bb056db36f81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:31:50.007060 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.006982 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c123b1-afa2-4560-9bf1-bb056db36f81-kube-api-access-xcc64" (OuterVolumeSpecName: "kube-api-access-xcc64") pod "08c123b1-afa2-4560-9bf1-bb056db36f81" (UID: "08c123b1-afa2-4560-9bf1-bb056db36f81"). InnerVolumeSpecName "kube-api-access-xcc64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:31:50.007125 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.007108 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08c123b1-afa2-4560-9bf1-bb056db36f81" (UID: "08c123b1-afa2-4560-9bf1-bb056db36f81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:31:50.007125 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.007111 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-token" (OuterVolumeSpecName: "istio-token") pod "08c123b1-afa2-4560-9bf1-bb056db36f81" (UID: "08c123b1-afa2-4560-9bf1-bb056db36f81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:31:50.106675 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.106640 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:50.106675 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.106668 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:50.106675 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.106678 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcc64\" (UniqueName: \"kubernetes.io/projected/08c123b1-afa2-4560-9bf1-bb056db36f81-kube-api-access-xcc64\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:50.106675 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.106687 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c123b1-afa2-4560-9bf1-bb056db36f81-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:31:50.882981 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.882950 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" Sep 11 14:31:50.883439 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.882953 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd" event={"ID":"08c123b1-afa2-4560-9bf1-bb056db36f81","Type":"ContainerDied","Data":"0caf8497dadb58ef1ed114c5127849087f0978d903f7ba0d4c0eec12ec6dd38b"} Sep 11 14:31:50.883439 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.883098 2573 scope.go:117] "RemoveContainer" containerID="e5568f1fe185f6c35721d760934d2296a933007c57574c71eab154d17a72103a" Sep 11 14:31:50.904594 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.904569 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd"] Sep 11 14:31:50.912506 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:50.912481 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a39864-openshift-default-85dc75cd6d-m96sd"] Sep 11 14:31:52.187114 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:52.187075 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08c123b1-afa2-4560-9bf1-bb056db36f81" path="/var/lib/kubelet/pods/08c123b1-afa2-4560-9bf1-bb056db36f81/volumes" Sep 11 14:31:53.626818 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:53.625871 2573 patch_prober.go:28] interesting pod/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": read tcp 10.132.0.2:38700->10.132.0.66:15021: read: connection reset by peer" start-of-body= Sep 11 14:31:53.626818 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:53.625934 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" podUID="fa1b2aea-b48f-4197-8386-c6072ad36889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": read tcp 10.132.0.2:38700->10.132.0.66:15021: read: connection reset by peer" Sep 11 14:31:53.904570 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:53.904490 2573 generic.go:356] "Generic (PLEG): container finished" podID="fa1b2aea-b48f-4197-8386-c6072ad36889" containerID="77deee2cb5bc3f44a6fd837a3ce91b543fb83dd45605523d8a891da2d3e19d04" exitCode=0 Sep 11 14:31:53.904737 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:53.904568 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" event={"ID":"fa1b2aea-b48f-4197-8386-c6072ad36889","Type":"ContainerDied","Data":"77deee2cb5bc3f44a6fd837a3ce91b543fb83dd45605523d8a891da2d3e19d04"} Sep 11 14:31:53.905049 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:53.905032 2573 scope.go:117] "RemoveContainer" containerID="77deee2cb5bc3f44a6fd837a3ce91b543fb83dd45605523d8a891da2d3e19d04" Sep 11 14:31:53.906121 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:53.906095 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:31:53.906225 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:53.906178 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:31:53.906289 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:53.906226 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:31:54.911641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:54.911603 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" event={"ID":"fa1b2aea-b48f-4197-8386-c6072ad36889","Type":"ContainerStarted","Data":"90894869f8aa2c9a0e2362d9a17590bb337e498321d758d4f36a218e9eebc3fc"} Sep 11 14:31:55.681130 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:55.681075 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:31:55.681352 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:55.681210 2573 patch_prober.go:28] interesting pod/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 14:31:55.681352 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:55.681267 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" podUID="fa1b2aea-b48f-4197-8386-c6072ad36889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 14:31:56.681570 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:56.681535 2573 patch_prober.go:28] interesting pod/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 14:31:56.682011 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:56.681595 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" podUID="fa1b2aea-b48f-4197-8386-c6072ad36889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 14:31:57.681209 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:57.681170 2573 patch_prober.go:28] interesting pod/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 14:31:57.681395 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:57.681237 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" podUID="fa1b2aea-b48f-4197-8386-c6072ad36889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 14:31:58.680989 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:58.680955 2573 patch_prober.go:28] interesting pod/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 14:31:58.681348 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:58.681015 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" podUID="fa1b2aea-b48f-4197-8386-c6072ad36889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 14:31:59.681141 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:59.681107 2573 patch_prober.go:28] interesting pod/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 14:31:59.681503 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:31:59.681168 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" podUID="fa1b2aea-b48f-4197-8386-c6072ad36889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 14:32:00.685144 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:00.685109 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:32:00.685513 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:00.685397 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:32:00.685997 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:00.685976 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:32:03.384507 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:03.384476 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm"] Sep 11 14:32:03.385080 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:03.384697 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" podUID="3d2a384c-8016-4611-b381-df0695e86d41" containerName="istio-proxy" containerID="cri-o://698bd5e12e2711963de6fe084b7dc26a6b2fd281c7e0826ff84506d7e9aee4db" gracePeriod=30 Sep 11 14:32:04.366176 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.366147 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd"] Sep 11 14:32:04.366754 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.366734 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerName="istio-proxy" Sep 11 14:32:04.366754 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.366753 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerName="istio-proxy" Sep 11 14:32:04.366930 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.366893 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerName="istio-proxy" Sep 11 14:32:04.366930 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.366908 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerName="istio-proxy" Sep 11 14:32:04.370518 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.370497 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.373189 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.373170 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-72e046-openshift-default-dockercfg-bfqvx\"" Sep 11 14:32:04.386717 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.386690 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd"] Sep 11 14:32:04.548722 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.548684 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ce241e3-382a-4991-b985-4ea068d2458b-istiod-ca-cert\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.549092 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.549064 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-workload-certs\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.549230 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.549211 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-istio-envoy\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.549338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.549288 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-istio-data\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.549338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.549320 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqjr4\" (UniqueName: \"kubernetes.io/projected/3ce241e3-382a-4991-b985-4ea068d2458b-kube-api-access-kqjr4\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.549454 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.549358 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-credential-socket\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.549454 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.549397 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ce241e3-382a-4991-b985-4ea068d2458b-istio-podinfo\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.549568 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.549455 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce241e3-382a-4991-b985-4ea068d2458b-istio-token\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.549568 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.549499 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-workload-socket\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.650299 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.650206 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ce241e3-382a-4991-b985-4ea068d2458b-istiod-ca-cert\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.650299 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.650287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-workload-certs\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.650537 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.650304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-istio-envoy\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.650537 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.650320 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-istio-data\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.650537 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.650335 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqjr4\" (UniqueName: \"kubernetes.io/projected/3ce241e3-382a-4991-b985-4ea068d2458b-kube-api-access-kqjr4\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.650537 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.650368 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-credential-socket\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.650537 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.650398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ce241e3-382a-4991-b985-4ea068d2458b-istio-podinfo\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.650537 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.650447 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce241e3-382a-4991-b985-4ea068d2458b-istio-token\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.650537 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.650486 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-workload-socket\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.650972 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.650733 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-istio-data\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.650972 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.650899 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-credential-socket\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.650972 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.650952 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-workload-socket\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.651125 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.651020 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-workload-certs\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.651125 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.651058 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ce241e3-382a-4991-b985-4ea068d2458b-istiod-ca-cert\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.652586 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.652567 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-istio-envoy\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.653024 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.653002 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ce241e3-382a-4991-b985-4ea068d2458b-istio-podinfo\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.660458 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.660434 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce241e3-382a-4991-b985-4ea068d2458b-istio-token\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.662002 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.661982 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqjr4\" (UniqueName: \"kubernetes.io/projected/3ce241e3-382a-4991-b985-4ea068d2458b-kube-api-access-kqjr4\") pod \"e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.683116 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.683095 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:04.969866 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.966733 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd"] Sep 11 14:32:04.975307 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.975269 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:32:04.975432 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.975352 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:32:04.975432 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:04.975396 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:32:05.989086 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:05.989039 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" event={"ID":"3ce241e3-382a-4991-b985-4ea068d2458b","Type":"ContainerStarted","Data":"8df9adf426b378ceb6f4dca22913a5fd0b91cfe52069094b559df50f88c2080c"} Sep 11 14:32:05.989530 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:05.989098 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" event={"ID":"3ce241e3-382a-4991-b985-4ea068d2458b","Type":"ContainerStarted","Data":"56bd048a994200e4721e228ac1bbe821b6bc9a385e17ba9b943ac4591528ec62"} Sep 11 14:32:06.015267 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:06.015205 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" podStartSLOduration=2.01515929 podStartE2EDuration="2.01515929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:32:06.011237724 +0000 UTC m=+1038.348205916" watchObservedRunningTime="2026-09-11 14:32:06.01515929 +0000 UTC m=+1038.352127481" Sep 11 14:32:06.683889 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:06.683823 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:06.689767 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:06.689742 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:06.994874 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:06.994757 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:06.995892 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:06.995871 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:08.627808 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.627782 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:32:08.795853 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.795753 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-workload-certs\") pod \"3d2a384c-8016-4611-b381-df0695e86d41\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " Sep 11 14:32:08.795853 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.795798 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-istio-envoy\") pod \"3d2a384c-8016-4611-b381-df0695e86d41\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " Sep 11 14:32:08.795853 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.795817 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2a384c-8016-4611-b381-df0695e86d41-istiod-ca-cert\") pod \"3d2a384c-8016-4611-b381-df0695e86d41\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " Sep 11 14:32:08.796116 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.795891 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-istio-data\") pod \"3d2a384c-8016-4611-b381-df0695e86d41\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " Sep 11 14:32:08.796116 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.795918 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2a384c-8016-4611-b381-df0695e86d41-istio-podinfo\") pod \"3d2a384c-8016-4611-b381-df0695e86d41\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " Sep 11 14:32:08.796116 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.795944 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2a384c-8016-4611-b381-df0695e86d41-istio-token\") pod \"3d2a384c-8016-4611-b381-df0695e86d41\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " Sep 11 14:32:08.796116 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.796073 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-credential-socket\") pod \"3d2a384c-8016-4611-b381-df0695e86d41\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " Sep 11 14:32:08.796116 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.796112 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbzpp\" (UniqueName: \"kubernetes.io/projected/3d2a384c-8016-4611-b381-df0695e86d41-kube-api-access-hbzpp\") pod \"3d2a384c-8016-4611-b381-df0695e86d41\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " Sep 11 14:32:08.796379 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.796073 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d2a384c-8016-4611-b381-df0695e86d41" (UID: "3d2a384c-8016-4611-b381-df0695e86d41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:08.796379 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.796185 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-workload-socket\") pod \"3d2a384c-8016-4611-b381-df0695e86d41\" (UID: \"3d2a384c-8016-4611-b381-df0695e86d41\") " Sep 11 14:32:08.796379 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.796303 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d2a384c-8016-4611-b381-df0695e86d41" (UID: "3d2a384c-8016-4611-b381-df0695e86d41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:08.796541 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.796408 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d2a384c-8016-4611-b381-df0695e86d41" (UID: "3d2a384c-8016-4611-b381-df0695e86d41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:08.796541 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.796162 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d2a384c-8016-4611-b381-df0695e86d41" (UID: "3d2a384c-8016-4611-b381-df0695e86d41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:08.796541 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.796433 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2a384c-8016-4611-b381-df0695e86d41-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d2a384c-8016-4611-b381-df0695e86d41" (UID: "3d2a384c-8016-4611-b381-df0695e86d41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:32:08.796702 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.796628 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:08.796702 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.796650 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:08.796702 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.796663 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2a384c-8016-4611-b381-df0695e86d41-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:08.796702 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.796677 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:08.796702 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.796690 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:08.798151 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.798122 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d2a384c-8016-4611-b381-df0695e86d41-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d2a384c-8016-4611-b381-df0695e86d41" (UID: "3d2a384c-8016-4611-b381-df0695e86d41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:32:08.798151 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.798119 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2a384c-8016-4611-b381-df0695e86d41-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d2a384c-8016-4611-b381-df0695e86d41" (UID: "3d2a384c-8016-4611-b381-df0695e86d41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:08.798306 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.798217 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2a384c-8016-4611-b381-df0695e86d41-kube-api-access-hbzpp" (OuterVolumeSpecName: "kube-api-access-hbzpp") pod "3d2a384c-8016-4611-b381-df0695e86d41" (UID: "3d2a384c-8016-4611-b381-df0695e86d41"). InnerVolumeSpecName "kube-api-access-hbzpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:08.798306 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.798250 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d2a384c-8016-4611-b381-df0695e86d41" (UID: "3d2a384c-8016-4611-b381-df0695e86d41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:08.897892 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.897864 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbzpp\" (UniqueName: \"kubernetes.io/projected/3d2a384c-8016-4611-b381-df0695e86d41-kube-api-access-hbzpp\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:08.897892 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.897889 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2a384c-8016-4611-b381-df0695e86d41-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:08.897892 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.897898 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2a384c-8016-4611-b381-df0695e86d41-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:08.898143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:08.897907 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2a384c-8016-4611-b381-df0695e86d41-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:09.008020 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:09.007990 2573 generic.go:356] "Generic (PLEG): container finished" podID="3d2a384c-8016-4611-b381-df0695e86d41" containerID="698bd5e12e2711963de6fe084b7dc26a6b2fd281c7e0826ff84506d7e9aee4db" exitCode=0 Sep 11 14:32:09.008185 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:09.008030 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" event={"ID":"3d2a384c-8016-4611-b381-df0695e86d41","Type":"ContainerDied","Data":"698bd5e12e2711963de6fe084b7dc26a6b2fd281c7e0826ff84506d7e9aee4db"} Sep 11 14:32:09.008185 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:09.008046 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" Sep 11 14:32:09.008185 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:09.008090 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm" event={"ID":"3d2a384c-8016-4611-b381-df0695e86d41","Type":"ContainerDied","Data":"2b772f99491d602b4f437ca675963e15479ea41cfb92badb00a5b16313465646"} Sep 11 14:32:09.008185 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:09.008118 2573 scope.go:117] "RemoveContainer" containerID="698bd5e12e2711963de6fe084b7dc26a6b2fd281c7e0826ff84506d7e9aee4db" Sep 11 14:32:09.018484 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:09.018467 2573 scope.go:117] "RemoveContainer" containerID="4d6103063d5115921f1e5c27aeb8396ed5f45b4fc5094c73e7e84a589081e688" Sep 11 14:32:09.027114 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:09.027096 2573 scope.go:117] "RemoveContainer" containerID="698bd5e12e2711963de6fe084b7dc26a6b2fd281c7e0826ff84506d7e9aee4db" Sep 11 14:32:09.027371 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:32:09.027355 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698bd5e12e2711963de6fe084b7dc26a6b2fd281c7e0826ff84506d7e9aee4db\": container with ID starting with 698bd5e12e2711963de6fe084b7dc26a6b2fd281c7e0826ff84506d7e9aee4db not found: ID does not exist" containerID="698bd5e12e2711963de6fe084b7dc26a6b2fd281c7e0826ff84506d7e9aee4db" Sep 11 14:32:09.027444 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:09.027379 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698bd5e12e2711963de6fe084b7dc26a6b2fd281c7e0826ff84506d7e9aee4db"} err="failed to get container status \"698bd5e12e2711963de6fe084b7dc26a6b2fd281c7e0826ff84506d7e9aee4db\": rpc error: code = NotFound desc = could not find container \"698bd5e12e2711963de6fe084b7dc26a6b2fd281c7e0826ff84506d7e9aee4db\": container with ID starting with 698bd5e12e2711963de6fe084b7dc26a6b2fd281c7e0826ff84506d7e9aee4db not found: ID does not exist" Sep 11 14:32:09.027444 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:09.027394 2573 scope.go:117] "RemoveContainer" containerID="4d6103063d5115921f1e5c27aeb8396ed5f45b4fc5094c73e7e84a589081e688" Sep 11 14:32:09.027640 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:32:09.027623 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d6103063d5115921f1e5c27aeb8396ed5f45b4fc5094c73e7e84a589081e688\": container with ID starting with 4d6103063d5115921f1e5c27aeb8396ed5f45b4fc5094c73e7e84a589081e688 not found: ID does not exist" containerID="4d6103063d5115921f1e5c27aeb8396ed5f45b4fc5094c73e7e84a589081e688" Sep 11 14:32:09.027674 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:09.027645 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d6103063d5115921f1e5c27aeb8396ed5f45b4fc5094c73e7e84a589081e688"} err="failed to get container status \"4d6103063d5115921f1e5c27aeb8396ed5f45b4fc5094c73e7e84a589081e688\": rpc error: code = NotFound desc = could not find container \"4d6103063d5115921f1e5c27aeb8396ed5f45b4fc5094c73e7e84a589081e688\": container with ID starting with 4d6103063d5115921f1e5c27aeb8396ed5f45b4fc5094c73e7e84a589081e688 not found: ID does not exist" Sep 11 14:32:09.033304 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:09.033278 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm"] Sep 11 14:32:09.035694 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:09.035672 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b6b864-openshift-default-5d598d5768-g4klm"] Sep 11 14:32:10.188720 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:10.188685 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d2a384c-8016-4611-b381-df0695e86d41" path="/var/lib/kubelet/pods/3d2a384c-8016-4611-b381-df0695e86d41/volumes" Sep 11 14:32:23.036993 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:23.036950 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 14:32:23.037499 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:23.037019 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" podUID="3ce241e3-382a-4991-b985-4ea068d2458b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 14:32:24.109768 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:24.109733 2573 generic.go:356] "Generic (PLEG): container finished" podID="3ce241e3-382a-4991-b985-4ea068d2458b" containerID="8df9adf426b378ceb6f4dca22913a5fd0b91cfe52069094b559df50f88c2080c" exitCode=0 Sep 11 14:32:24.110311 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:24.109808 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" event={"ID":"3ce241e3-382a-4991-b985-4ea068d2458b","Type":"ContainerDied","Data":"8df9adf426b378ceb6f4dca22913a5fd0b91cfe52069094b559df50f88c2080c"} Sep 11 14:32:24.110311 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:24.110273 2573 scope.go:117] "RemoveContainer" containerID="8df9adf426b378ceb6f4dca22913a5fd0b91cfe52069094b559df50f88c2080c" Sep 11 14:32:24.111340 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:24.111315 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:32:24.111398 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:24.111381 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:32:24.111442 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:24.111420 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:32:25.117618 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:25.117585 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" event={"ID":"3ce241e3-382a-4991-b985-4ea068d2458b","Type":"ContainerStarted","Data":"28b5ff03fb8da65c422aa7ca1f991870ed13616e344e536787051db34a6d0906"} Sep 11 14:32:25.684132 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:25.684101 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:25.684333 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:25.684239 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 11 14:32:25.684333 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:25.684290 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" podUID="3ce241e3-382a-4991-b985-4ea068d2458b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 11 14:32:26.684419 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:26.684389 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 11 14:32:26.684815 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:26.684443 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" podUID="3ce241e3-382a-4991-b985-4ea068d2458b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 11 14:32:27.684180 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:27.684146 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 11 14:32:27.684354 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:27.684213 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" podUID="3ce241e3-382a-4991-b985-4ea068d2458b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 11 14:32:28.683809 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.683771 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 11 14:32:28.684294 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.683868 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" podUID="3ce241e3-382a-4991-b985-4ea068d2458b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 11 14:32:28.923942 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.923906 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb"] Sep 11 14:32:28.924517 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.924500 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2a384c-8016-4611-b381-df0695e86d41" containerName="istio-proxy" Sep 11 14:32:28.924517 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.924518 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2a384c-8016-4611-b381-df0695e86d41" containerName="istio-proxy" Sep 11 14:32:28.924634 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.924538 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2a384c-8016-4611-b381-df0695e86d41" containerName="istio-proxy" Sep 11 14:32:28.924634 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.924547 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2a384c-8016-4611-b381-df0695e86d41" containerName="istio-proxy" Sep 11 14:32:28.924634 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.924570 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerName="istio-proxy" Sep 11 14:32:28.924634 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.924579 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c123b1-afa2-4560-9bf1-bb056db36f81" containerName="istio-proxy" Sep 11 14:32:28.924759 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.924682 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d2a384c-8016-4611-b381-df0695e86d41" containerName="istio-proxy" Sep 11 14:32:28.924759 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.924696 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d2a384c-8016-4611-b381-df0695e86d41" containerName="istio-proxy" Sep 11 14:32:28.946211 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.946138 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:28.951193 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.951172 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-344045-openshift-default-dockercfg-hxbgr\"" Sep 11 14:32:28.954920 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.954894 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb"] Sep 11 14:32:28.996028 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.996002 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73707577-c5a8-410d-b619-6d3cbf8a7724-istiod-ca-cert\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:28.996145 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.996037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-workload-socket\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:28.996145 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.996074 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-credential-socket\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:28.996223 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.996138 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-envoy\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:28.996270 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.996239 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-podinfo\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:28.996270 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.996259 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-token\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:28.996340 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.996291 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-workload-certs\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:28.996340 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.996327 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-data\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:28.996407 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:28.996362 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6txd7\" (UniqueName: \"kubernetes.io/projected/73707577-c5a8-410d-b619-6d3cbf8a7724-kube-api-access-6txd7\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.097225 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.097188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-credential-socket\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.097225 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.097226 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-envoy\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.097448 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.097296 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-podinfo\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.097448 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.097320 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-token\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.097448 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.097349 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-workload-certs\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.097448 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.097386 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-data\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.097448 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.097419 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6txd7\" (UniqueName: \"kubernetes.io/projected/73707577-c5a8-410d-b619-6d3cbf8a7724-kube-api-access-6txd7\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.097717 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.097490 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73707577-c5a8-410d-b619-6d3cbf8a7724-istiod-ca-cert\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.097717 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.097519 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-workload-socket\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.097845 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.097806 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-workload-certs\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.097909 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.097862 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-data\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.097951 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.097902 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-workload-socket\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.098074 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.098048 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-credential-socket\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.098286 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.098265 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73707577-c5a8-410d-b619-6d3cbf8a7724-istiod-ca-cert\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.099716 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.099689 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-envoy\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.099918 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.099897 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-podinfo\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.109017 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.108995 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-token\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.109461 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.109440 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6txd7\" (UniqueName: \"kubernetes.io/projected/73707577-c5a8-410d-b619-6d3cbf8a7724-kube-api-access-6txd7\") pod \"e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.264041 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.263948 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:29.496876 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.496823 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv"] Sep 11 14:32:29.497069 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.497033 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" podUID="fa1b2aea-b48f-4197-8386-c6072ad36889" containerName="istio-proxy" containerID="cri-o://90894869f8aa2c9a0e2362d9a17590bb337e498321d758d4f36a218e9eebc3fc" gracePeriod=30 Sep 11 14:32:29.566619 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.566592 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb"] Sep 11 14:32:29.567752 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:32:29.567722 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73707577_c5a8_410d_b619_6d3cbf8a7724.slice/crio-57deba62532fa61356bf74ed33bf43fd041259dd45bf9a4b1900b2060c8e06ad WatchSource:0}: Error finding container 57deba62532fa61356bf74ed33bf43fd041259dd45bf9a4b1900b2060c8e06ad: Status 404 returned error can't find the container with id 57deba62532fa61356bf74ed33bf43fd041259dd45bf9a4b1900b2060c8e06ad Sep 11 14:32:29.570161 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.570130 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:32:29.570253 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.570193 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:32:29.570253 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.570225 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:32:29.684109 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.684070 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 11 14:32:29.684504 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:29.684123 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" podUID="3ce241e3-382a-4991-b985-4ea068d2458b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 11 14:32:30.159870 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:30.159813 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" event={"ID":"73707577-c5a8-410d-b619-6d3cbf8a7724","Type":"ContainerStarted","Data":"2db4db4952fe59ab0d99ef6b276c505795fcd7d8787d31d882bcd286e7c74fce"} Sep 11 14:32:30.160076 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:30.159902 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" event={"ID":"73707577-c5a8-410d-b619-6d3cbf8a7724","Type":"ContainerStarted","Data":"57deba62532fa61356bf74ed33bf43fd041259dd45bf9a4b1900b2060c8e06ad"} Sep 11 14:32:30.218056 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:30.218007 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" podStartSLOduration=2.217991381 podStartE2EDuration="2.217991381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:32:30.216167591 +0000 UTC m=+1062.553135804" watchObservedRunningTime="2026-09-11 14:32:30.217991381 +0000 UTC m=+1062.554959571" Sep 11 14:32:30.264391 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:30.264355 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:30.268949 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:30.268918 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:30.687639 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:30.687609 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:30.688093 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:30.687820 2573 patch_prober.go:28] interesting pod/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 11 14:32:30.688093 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:30.687895 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" podUID="fa1b2aea-b48f-4197-8386-c6072ad36889" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:32:30.688093 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:30.688040 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:30.688685 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:30.688669 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:32:31.166270 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:31.166235 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:31.166921 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:31.166897 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:34.749960 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.749937 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:32:34.851380 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.851356 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-podinfo\") pod \"fa1b2aea-b48f-4197-8386-c6072ad36889\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " Sep 11 14:32:34.851516 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.851413 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-token\") pod \"fa1b2aea-b48f-4197-8386-c6072ad36889\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " Sep 11 14:32:34.851516 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.851444 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-workload-certs\") pod \"fa1b2aea-b48f-4197-8386-c6072ad36889\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " Sep 11 14:32:34.851516 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.851464 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkww7\" (UniqueName: \"kubernetes.io/projected/fa1b2aea-b48f-4197-8386-c6072ad36889-kube-api-access-vkww7\") pod \"fa1b2aea-b48f-4197-8386-c6072ad36889\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " Sep 11 14:32:34.851516 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.851498 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa1b2aea-b48f-4197-8386-c6072ad36889-istiod-ca-cert\") pod \"fa1b2aea-b48f-4197-8386-c6072ad36889\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " Sep 11 14:32:34.851871 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.851582 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-credential-socket\") pod \"fa1b2aea-b48f-4197-8386-c6072ad36889\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " Sep 11 14:32:34.851871 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.851666 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-envoy\") pod \"fa1b2aea-b48f-4197-8386-c6072ad36889\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " Sep 11 14:32:34.851871 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.851696 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-data\") pod \"fa1b2aea-b48f-4197-8386-c6072ad36889\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " Sep 11 14:32:34.851871 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.851732 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-workload-socket\") pod \"fa1b2aea-b48f-4197-8386-c6072ad36889\" (UID: \"fa1b2aea-b48f-4197-8386-c6072ad36889\") " Sep 11 14:32:34.851871 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.851737 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fa1b2aea-b48f-4197-8386-c6072ad36889" (UID: "fa1b2aea-b48f-4197-8386-c6072ad36889"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:34.851871 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.851824 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fa1b2aea-b48f-4197-8386-c6072ad36889" (UID: "fa1b2aea-b48f-4197-8386-c6072ad36889"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:34.852190 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.851906 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1b2aea-b48f-4197-8386-c6072ad36889-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fa1b2aea-b48f-4197-8386-c6072ad36889" (UID: "fa1b2aea-b48f-4197-8386-c6072ad36889"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:32:34.852190 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.852050 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-data" (OuterVolumeSpecName: "istio-data") pod "fa1b2aea-b48f-4197-8386-c6072ad36889" (UID: "fa1b2aea-b48f-4197-8386-c6072ad36889"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:34.852190 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.852113 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fa1b2aea-b48f-4197-8386-c6072ad36889" (UID: "fa1b2aea-b48f-4197-8386-c6072ad36889"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:34.852352 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.852273 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:34.852352 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.852291 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa1b2aea-b48f-4197-8386-c6072ad36889-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:34.852352 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.852304 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:34.852352 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.852318 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:34.852352 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.852332 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:34.853613 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.853578 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-token" (OuterVolumeSpecName: "istio-token") pod "fa1b2aea-b48f-4197-8386-c6072ad36889" (UID: "fa1b2aea-b48f-4197-8386-c6072ad36889"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:34.853733 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.853715 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fa1b2aea-b48f-4197-8386-c6072ad36889" (UID: "fa1b2aea-b48f-4197-8386-c6072ad36889"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:34.853967 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.853949 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fa1b2aea-b48f-4197-8386-c6072ad36889" (UID: "fa1b2aea-b48f-4197-8386-c6072ad36889"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:32:34.854035 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.853982 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1b2aea-b48f-4197-8386-c6072ad36889-kube-api-access-vkww7" (OuterVolumeSpecName: "kube-api-access-vkww7") pod "fa1b2aea-b48f-4197-8386-c6072ad36889" (UID: "fa1b2aea-b48f-4197-8386-c6072ad36889"). InnerVolumeSpecName "kube-api-access-vkww7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:34.952784 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.952753 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:34.952784 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.952778 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:34.952981 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.952791 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa1b2aea-b48f-4197-8386-c6072ad36889-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:34.952981 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:34.952804 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkww7\" (UniqueName: \"kubernetes.io/projected/fa1b2aea-b48f-4197-8386-c6072ad36889-kube-api-access-vkww7\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:32:35.193548 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:35.193469 2573 generic.go:356] "Generic (PLEG): container finished" podID="fa1b2aea-b48f-4197-8386-c6072ad36889" containerID="90894869f8aa2c9a0e2362d9a17590bb337e498321d758d4f36a218e9eebc3fc" exitCode=0 Sep 11 14:32:35.193548 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:35.193527 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" Sep 11 14:32:35.193726 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:35.193546 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" event={"ID":"fa1b2aea-b48f-4197-8386-c6072ad36889","Type":"ContainerDied","Data":"90894869f8aa2c9a0e2362d9a17590bb337e498321d758d4f36a218e9eebc3fc"} Sep 11 14:32:35.193726 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:35.193594 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv" event={"ID":"fa1b2aea-b48f-4197-8386-c6072ad36889","Type":"ContainerDied","Data":"a6aafdf92076da8ceab8ac0ed1b7a6d3766ca420f748241f6296ee777c656ca4"} Sep 11 14:32:35.193726 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:35.193617 2573 scope.go:117] "RemoveContainer" containerID="90894869f8aa2c9a0e2362d9a17590bb337e498321d758d4f36a218e9eebc3fc" Sep 11 14:32:35.203136 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:35.203120 2573 scope.go:117] "RemoveContainer" containerID="77deee2cb5bc3f44a6fd837a3ce91b543fb83dd45605523d8a891da2d3e19d04" Sep 11 14:32:35.211553 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:35.211536 2573 scope.go:117] "RemoveContainer" containerID="90894869f8aa2c9a0e2362d9a17590bb337e498321d758d4f36a218e9eebc3fc" Sep 11 14:32:35.211827 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:32:35.211807 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90894869f8aa2c9a0e2362d9a17590bb337e498321d758d4f36a218e9eebc3fc\": container with ID starting with 90894869f8aa2c9a0e2362d9a17590bb337e498321d758d4f36a218e9eebc3fc not found: ID does not exist" containerID="90894869f8aa2c9a0e2362d9a17590bb337e498321d758d4f36a218e9eebc3fc" Sep 11 14:32:35.211919 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:35.211849 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90894869f8aa2c9a0e2362d9a17590bb337e498321d758d4f36a218e9eebc3fc"} err="failed to get container status \"90894869f8aa2c9a0e2362d9a17590bb337e498321d758d4f36a218e9eebc3fc\": rpc error: code = NotFound desc = could not find container \"90894869f8aa2c9a0e2362d9a17590bb337e498321d758d4f36a218e9eebc3fc\": container with ID starting with 90894869f8aa2c9a0e2362d9a17590bb337e498321d758d4f36a218e9eebc3fc not found: ID does not exist" Sep 11 14:32:35.211919 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:35.211867 2573 scope.go:117] "RemoveContainer" containerID="77deee2cb5bc3f44a6fd837a3ce91b543fb83dd45605523d8a891da2d3e19d04" Sep 11 14:32:35.218668 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:32:35.212446 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77deee2cb5bc3f44a6fd837a3ce91b543fb83dd45605523d8a891da2d3e19d04\": container with ID starting with 77deee2cb5bc3f44a6fd837a3ce91b543fb83dd45605523d8a891da2d3e19d04 not found: ID does not exist" containerID="77deee2cb5bc3f44a6fd837a3ce91b543fb83dd45605523d8a891da2d3e19d04" Sep 11 14:32:35.218668 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:35.212483 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77deee2cb5bc3f44a6fd837a3ce91b543fb83dd45605523d8a891da2d3e19d04"} err="failed to get container status \"77deee2cb5bc3f44a6fd837a3ce91b543fb83dd45605523d8a891da2d3e19d04\": rpc error: code = NotFound desc = could not find container \"77deee2cb5bc3f44a6fd837a3ce91b543fb83dd45605523d8a891da2d3e19d04\": container with ID starting with 77deee2cb5bc3f44a6fd837a3ce91b543fb83dd45605523d8a891da2d3e19d04 not found: ID does not exist" Sep 11 14:32:35.227940 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:35.227917 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv"] Sep 11 14:32:35.233956 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:35.233933 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-02723795-openshift-default-7ff745bfcb-fmqkv"] Sep 11 14:32:36.188032 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:36.187997 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa1b2aea-b48f-4197-8386-c6072ad36889" path="/var/lib/kubelet/pods/fa1b2aea-b48f-4197-8386-c6072ad36889/volumes" Sep 11 14:32:45.877533 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.877486 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk"] Sep 11 14:32:45.878057 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.877964 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa1b2aea-b48f-4197-8386-c6072ad36889" containerName="istio-proxy" Sep 11 14:32:45.878057 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.877981 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa1b2aea-b48f-4197-8386-c6072ad36889" containerName="istio-proxy" Sep 11 14:32:45.878057 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.878013 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa1b2aea-b48f-4197-8386-c6072ad36889" containerName="istio-proxy" Sep 11 14:32:45.878057 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.878022 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa1b2aea-b48f-4197-8386-c6072ad36889" containerName="istio-proxy" Sep 11 14:32:45.878204 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.878108 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa1b2aea-b48f-4197-8386-c6072ad36889" containerName="istio-proxy" Sep 11 14:32:45.878204 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.878121 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa1b2aea-b48f-4197-8386-c6072ad36889" containerName="istio-proxy" Sep 11 14:32:45.881167 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.881151 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:45.884117 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.884095 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b934e8-openshift-default-dockercfg-nr5r7\"" Sep 11 14:32:45.896304 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.896278 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk"] Sep 11 14:32:45.963227 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.963197 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-workload-socket\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:45.963426 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.963269 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-credential-socket\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:45.963426 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.963305 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-podinfo\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:45.963426 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.963325 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-data\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:45.963426 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.963375 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-envoy\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:45.963426 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.963406 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bghb2\" (UniqueName: \"kubernetes.io/projected/1e9027ad-28a5-44d9-b935-fcf83bee47de-kube-api-access-bghb2\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:45.963713 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.963435 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e9027ad-28a5-44d9-b935-fcf83bee47de-istiod-ca-cert\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:45.963713 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.963537 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-workload-certs\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:45.963713 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:45.963652 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-token\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.064475 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.064436 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-envoy\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.064475 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.064480 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghb2\" (UniqueName: \"kubernetes.io/projected/1e9027ad-28a5-44d9-b935-fcf83bee47de-kube-api-access-bghb2\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.064707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.064508 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e9027ad-28a5-44d9-b935-fcf83bee47de-istiod-ca-cert\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.064707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.064533 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-workload-certs\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.064707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.064572 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-token\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.064707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.064605 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-workload-socket\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.064707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.064678 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-credential-socket\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.065018 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.064720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-podinfo\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.065018 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.064745 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-data\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.065133 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.065070 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-workload-certs\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.065329 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.065284 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e9027ad-28a5-44d9-b935-fcf83bee47de-istiod-ca-cert\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.065464 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.065442 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-workload-socket\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.065519 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.065467 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-credential-socket\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.065519 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.065482 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-data\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.066934 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.066908 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-envoy\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.067682 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.067663 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-podinfo\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.077501 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.077454 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-token\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.077610 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.077593 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghb2\" (UniqueName: \"kubernetes.io/projected/1e9027ad-28a5-44d9-b935-fcf83bee47de-kube-api-access-bghb2\") pod \"e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.193362 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.193284 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:46.520339 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:32:46.520295 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9027ad_28a5_44d9_b935_fcf83bee47de.slice/crio-c33009ffa480f2f3bb15eadd51417e190537ab06ec741180f449b5983b501fdc WatchSource:0}: Error finding container c33009ffa480f2f3bb15eadd51417e190537ab06ec741180f449b5983b501fdc: Status 404 returned error can't find the container with id c33009ffa480f2f3bb15eadd51417e190537ab06ec741180f449b5983b501fdc Sep 11 14:32:46.522164 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.522070 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk"] Sep 11 14:32:46.522940 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.522900 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:32:46.523033 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.522989 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:32:46.523087 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:46.523069 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:32:47.014632 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:47.014359 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": read tcp 10.132.0.2:56132->10.132.0.68:15021: read: connection reset by peer" start-of-body= Sep 11 14:32:47.014632 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:47.014429 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" podUID="73707577-c5a8-410d-b619-6d3cbf8a7724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": read tcp 10.132.0.2:56132->10.132.0.68:15021: read: connection reset by peer" Sep 11 14:32:47.278502 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:47.278377 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" event={"ID":"1e9027ad-28a5-44d9-b935-fcf83bee47de","Type":"ContainerStarted","Data":"83a99cbd84aea532d0d03b55fd632dd0b3ad6f2b822582d3e3bfd7f01eff75f3"} Sep 11 14:32:47.278502 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:47.278472 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" event={"ID":"1e9027ad-28a5-44d9-b935-fcf83bee47de","Type":"ContainerStarted","Data":"c33009ffa480f2f3bb15eadd51417e190537ab06ec741180f449b5983b501fdc"} Sep 11 14:32:47.280069 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:47.280042 2573 generic.go:356] "Generic (PLEG): container finished" podID="73707577-c5a8-410d-b619-6d3cbf8a7724" containerID="2db4db4952fe59ab0d99ef6b276c505795fcd7d8787d31d882bcd286e7c74fce" exitCode=0 Sep 11 14:32:47.280183 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:47.280091 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" event={"ID":"73707577-c5a8-410d-b619-6d3cbf8a7724","Type":"ContainerDied","Data":"2db4db4952fe59ab0d99ef6b276c505795fcd7d8787d31d882bcd286e7c74fce"} Sep 11 14:32:47.280433 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:47.280420 2573 scope.go:117] "RemoveContainer" containerID="2db4db4952fe59ab0d99ef6b276c505795fcd7d8787d31d882bcd286e7c74fce" Sep 11 14:32:47.281411 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:47.281380 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:32:47.281506 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:47.281447 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:32:47.281506 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:47.281496 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:32:47.332940 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:47.332894 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" podStartSLOduration=2.332879641 podStartE2EDuration="2.332879641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:32:47.331795962 +0000 UTC m=+1079.668764153" watchObservedRunningTime="2026-09-11 14:32:47.332879641 +0000 UTC m=+1079.669847831" Sep 11 14:32:48.193861 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:48.193816 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:48.198916 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:48.198893 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:48.288370 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:48.288335 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" event={"ID":"73707577-c5a8-410d-b619-6d3cbf8a7724","Type":"ContainerStarted","Data":"b57ee078c3dffd8a7c4ccc24c3bb96ed03d4cfc157ac63362dcc53bad672f0ae"} Sep 11 14:32:48.288688 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:48.288665 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:48.289571 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:48.289550 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:32:49.264385 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:49.264349 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:49.264810 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:49.264678 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:32:49.264810 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:49.264734 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" podUID="73707577-c5a8-410d-b619-6d3cbf8a7724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 11 14:32:50.264663 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:50.264622 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:32:50.265098 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:50.264686 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" podUID="73707577-c5a8-410d-b619-6d3cbf8a7724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 11 14:32:51.264723 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:51.264680 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:32:51.265156 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:51.264741 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" podUID="73707577-c5a8-410d-b619-6d3cbf8a7724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 11 14:32:52.264750 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:52.264710 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:32:52.265172 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:52.264770 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" podUID="73707577-c5a8-410d-b619-6d3cbf8a7724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 11 14:32:53.264746 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:53.264709 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:32:53.264948 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:53.264770 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" podUID="73707577-c5a8-410d-b619-6d3cbf8a7724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 11 14:32:54.267841 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:54.267799 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:54.268260 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:54.268165 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:32:54.268960 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:32:54.268939 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:33:04.329999 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:04.329955 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 14:33:04.330363 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:04.330024 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" podUID="1e9027ad-28a5-44d9-b935-fcf83bee47de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 14:33:06.408579 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:06.408546 2573 generic.go:356] "Generic (PLEG): container finished" podID="1e9027ad-28a5-44d9-b935-fcf83bee47de" containerID="83a99cbd84aea532d0d03b55fd632dd0b3ad6f2b822582d3e3bfd7f01eff75f3" exitCode=0 Sep 11 14:33:06.409031 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:06.408604 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" event={"ID":"1e9027ad-28a5-44d9-b935-fcf83bee47de","Type":"ContainerDied","Data":"83a99cbd84aea532d0d03b55fd632dd0b3ad6f2b822582d3e3bfd7f01eff75f3"} Sep 11 14:33:06.409090 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:06.409070 2573 scope.go:117] "RemoveContainer" containerID="83a99cbd84aea532d0d03b55fd632dd0b3ad6f2b822582d3e3bfd7f01eff75f3" Sep 11 14:33:06.410173 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:06.410145 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:33:06.410243 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:06.410210 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:33:06.410243 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:06.410240 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:33:07.417811 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:07.417776 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" event={"ID":"1e9027ad-28a5-44d9-b935-fcf83bee47de","Type":"ContainerStarted","Data":"01768a1d5ffff7fc6c3c1cb5decb2b8234918a0c2dd6dfa18b516928044bf058"} Sep 11 14:33:08.193657 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:08.193625 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:33:08.193863 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:08.193824 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 14:33:08.193926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:08.193889 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" podUID="1e9027ad-28a5-44d9-b935-fcf83bee47de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 14:33:09.193812 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:09.193777 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 14:33:09.194214 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:09.193855 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" podUID="1e9027ad-28a5-44d9-b935-fcf83bee47de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 14:33:10.194093 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:10.194047 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 14:33:10.194580 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:10.194124 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" podUID="1e9027ad-28a5-44d9-b935-fcf83bee47de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 14:33:11.194242 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:11.194198 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 14:33:11.194616 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:11.194262 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" podUID="1e9027ad-28a5-44d9-b935-fcf83bee47de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 14:33:12.194062 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:12.194025 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 14:33:12.194237 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:12.194080 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" podUID="1e9027ad-28a5-44d9-b935-fcf83bee47de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 14:33:13.197686 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:13.197654 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:33:13.198098 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:13.197930 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:33:13.198671 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:13.198655 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:33:15.331492 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.331455 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k"] Sep 11 14:33:15.335715 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.335689 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.338657 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.338635 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-cd8e4a-openshift-default-dockercfg-srsvc\"" Sep 11 14:33:15.348683 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.348660 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k"] Sep 11 14:33:15.443312 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.443275 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-podinfo\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.443312 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.443316 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-token\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.443545 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.443372 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-envoy\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.443545 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.443421 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-credential-socket\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.443545 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.443463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdvd5\" (UniqueName: \"kubernetes.io/projected/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-kube-api-access-xdvd5\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.443545 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.443487 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istiod-ca-cert\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.443545 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.443521 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-workload-certs\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.443545 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.443537 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-workload-socket\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.443818 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.443645 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-data\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.544471 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.544435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-podinfo\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.544648 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.544484 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-token\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.544648 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.544521 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-envoy\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.544648 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.544548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-credential-socket\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.544648 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.544579 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdvd5\" (UniqueName: \"kubernetes.io/projected/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-kube-api-access-xdvd5\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.544648 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.544610 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istiod-ca-cert\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.544959 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.544663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-workload-certs\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.544959 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.544753 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-workload-socket\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.544959 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.544844 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-data\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.545131 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.545104 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-credential-socket\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.545203 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.545147 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-workload-certs\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.545262 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.545242 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-workload-socket\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.545345 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.545324 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-data\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.545591 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.545573 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istiod-ca-cert\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.547020 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.547001 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-envoy\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.547309 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.547280 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-podinfo\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.557422 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.557398 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdvd5\" (UniqueName: \"kubernetes.io/projected/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-kube-api-access-xdvd5\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.557580 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.557557 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-token\") pod \"e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.647909 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.647813 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:15.927815 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.927787 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k"] Sep 11 14:33:15.929947 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:33:15.929916 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ba867f_8e72_49f6_8f27_7cf8d0411f48.slice/crio-47a5d3c01af85b84c459ad28958d2b87d22a0aa04973a2a9313ff59a852d20df WatchSource:0}: Error finding container 47a5d3c01af85b84c459ad28958d2b87d22a0aa04973a2a9313ff59a852d20df: Status 404 returned error can't find the container with id 47a5d3c01af85b84c459ad28958d2b87d22a0aa04973a2a9313ff59a852d20df Sep 11 14:33:15.932133 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.932096 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:33:15.932240 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.932175 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:33:15.932240 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:15.932216 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:33:16.483600 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:16.483558 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" event={"ID":"d9ba867f-8e72-49f6-8f27-7cf8d0411f48","Type":"ContainerStarted","Data":"9951135d36f3293355ddebab627b0f928001da54a3be96d2f1708d3fd6090aa2"} Sep 11 14:33:16.484004 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:16.483620 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" event={"ID":"d9ba867f-8e72-49f6-8f27-7cf8d0411f48","Type":"ContainerStarted","Data":"47a5d3c01af85b84c459ad28958d2b87d22a0aa04973a2a9313ff59a852d20df"} Sep 11 14:33:16.507437 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:16.507375 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" podStartSLOduration=1.5073586460000001 podStartE2EDuration="1.507358646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:16.50519575 +0000 UTC m=+1108.842163941" watchObservedRunningTime="2026-09-11 14:33:16.507358646 +0000 UTC m=+1108.844326837" Sep 11 14:33:16.648324 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:16.648288 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:16.653155 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:16.653129 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:17.490122 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:17.490088 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:17.491111 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:17.491091 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:38.622151 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:38.622119 2573 generic.go:356] "Generic (PLEG): container finished" podID="d9ba867f-8e72-49f6-8f27-7cf8d0411f48" containerID="9951135d36f3293355ddebab627b0f928001da54a3be96d2f1708d3fd6090aa2" exitCode=0 Sep 11 14:33:38.622564 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:38.622182 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" event={"ID":"d9ba867f-8e72-49f6-8f27-7cf8d0411f48","Type":"ContainerDied","Data":"9951135d36f3293355ddebab627b0f928001da54a3be96d2f1708d3fd6090aa2"} Sep 11 14:33:38.622564 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:38.622561 2573 scope.go:117] "RemoveContainer" containerID="9951135d36f3293355ddebab627b0f928001da54a3be96d2f1708d3fd6090aa2" Sep 11 14:33:38.623627 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:38.623598 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:33:38.623728 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:38.623665 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:33:38.623728 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:38.623698 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:33:39.633455 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:39.633410 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" event={"ID":"d9ba867f-8e72-49f6-8f27-7cf8d0411f48","Type":"ContainerStarted","Data":"988e62bb18bc28871ad9ad7f8c84dc8f25f413ee81191fe98a9fe172726b8d36"} Sep 11 14:33:39.649280 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:39.649246 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:39.649558 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:39.649293 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:33:39.649728 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:39.649706 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" podUID="d9ba867f-8e72-49f6-8f27-7cf8d0411f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 11 14:33:40.649066 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:40.649029 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:33:40.649644 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:40.649610 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" podUID="d9ba867f-8e72-49f6-8f27-7cf8d0411f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 11 14:33:41.650304 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:41.650266 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:33:41.650717 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:41.650337 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" podUID="d9ba867f-8e72-49f6-8f27-7cf8d0411f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 11 14:33:42.648261 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:42.648215 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:33:42.648472 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:42.648287 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" podUID="d9ba867f-8e72-49f6-8f27-7cf8d0411f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 11 14:33:43.648401 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:43.648364 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:33:43.649024 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:43.648984 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" podUID="d9ba867f-8e72-49f6-8f27-7cf8d0411f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 11 14:33:44.648606 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:44.648565 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:33:44.649109 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:44.648629 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" podUID="d9ba867f-8e72-49f6-8f27-7cf8d0411f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 11 14:33:45.652156 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:45.652124 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:45.652593 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:45.652570 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:45.653252 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:45.653233 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:33:46.186971 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:46.186940 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7773-81898e"] Sep 11 14:33:46.189554 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:46.189530 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-81898e" Sep 11 14:33:46.192061 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:46.192043 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-998c2\"" Sep 11 14:33:46.193760 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:46.193740 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7773-81898e"] Sep 11 14:33:46.264206 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:46.264168 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz87r\" (UniqueName: \"kubernetes.io/projected/2a074075-b43b-42e5-94f9-c9056506a579-kube-api-access-cz87r\") pod \"test-curl-7773-81898e\" (UID: \"2a074075-b43b-42e5-94f9-c9056506a579\") " pod="opendatahub/test-curl-7773-81898e" Sep 11 14:33:46.364884 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:46.364822 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz87r\" (UniqueName: \"kubernetes.io/projected/2a074075-b43b-42e5-94f9-c9056506a579-kube-api-access-cz87r\") pod \"test-curl-7773-81898e\" (UID: \"2a074075-b43b-42e5-94f9-c9056506a579\") " pod="opendatahub/test-curl-7773-81898e" Sep 11 14:33:46.377270 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:46.377242 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz87r\" (UniqueName: \"kubernetes.io/projected/2a074075-b43b-42e5-94f9-c9056506a579-kube-api-access-cz87r\") pod \"test-curl-7773-81898e\" (UID: \"2a074075-b43b-42e5-94f9-c9056506a579\") " pod="opendatahub/test-curl-7773-81898e" Sep 11 14:33:46.502108 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:46.502026 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-81898e" Sep 11 14:33:46.777019 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:46.776993 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7773-81898e"] Sep 11 14:33:46.778282 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:33:46.778256 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a074075_b43b_42e5_94f9_c9056506a579.slice/crio-1d5d1a7706e2705940ab00ae39b218307309201cadb7b2dcceb4c982cd222514 WatchSource:0}: Error finding container 1d5d1a7706e2705940ab00ae39b218307309201cadb7b2dcceb4c982cd222514: Status 404 returned error can't find the container with id 1d5d1a7706e2705940ab00ae39b218307309201cadb7b2dcceb4c982cd222514 Sep 11 14:33:47.696363 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:47.696316 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7773-81898e" event={"ID":"2a074075-b43b-42e5-94f9-c9056506a579","Type":"ContainerStarted","Data":"91928ebd112036fffd13fb5c99eb8db9750edbd60b17592d758ada95de633f64"} Sep 11 14:33:47.696363 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:47.696370 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7773-81898e" event={"ID":"2a074075-b43b-42e5-94f9-c9056506a579","Type":"ContainerStarted","Data":"1d5d1a7706e2705940ab00ae39b218307309201cadb7b2dcceb4c982cd222514"} Sep 11 14:33:47.715693 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:47.715643 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7773-81898e" podStartSLOduration=1.715629633 podStartE2EDuration="1.715629633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:47.712692195 +0000 UTC m=+1140.049660387" watchObservedRunningTime="2026-09-11 14:33:47.715629633 +0000 UTC m=+1140.052597866" Sep 11 14:33:48.499733 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:48.499647 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7773-81898e"] Sep 11 14:33:48.506179 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:48.506152 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7773-81898e"] Sep 11 14:33:48.660550 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:48.660511 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7773-9d53ec"] Sep 11 14:33:48.661105 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:48.661077 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a074075-b43b-42e5-94f9-c9056506a579" containerName="test-curl-7773-81898e" Sep 11 14:33:48.661105 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:48.661102 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a074075-b43b-42e5-94f9-c9056506a579" containerName="test-curl-7773-81898e" Sep 11 14:33:48.661289 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:48.661210 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a074075-b43b-42e5-94f9-c9056506a579" containerName="test-curl-7773-81898e" Sep 11 14:33:48.663597 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:48.663576 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-9d53ec" Sep 11 14:33:48.673165 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:48.673141 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7773-9d53ec"] Sep 11 14:33:48.687227 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:48.687199 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq5rf\" (UniqueName: \"kubernetes.io/projected/0c8616f5-2fcb-44c0-b493-ceee84a09e1f-kube-api-access-rq5rf\") pod \"test-curl-7773-9d53ec\" (UID: \"0c8616f5-2fcb-44c0-b493-ceee84a09e1f\") " pod="opendatahub/test-curl-7773-9d53ec" Sep 11 14:33:48.702565 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:48.702544 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7773-81898e" container="test-curl-7773-81898e" Sep 11 14:33:48.788721 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:48.788645 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq5rf\" (UniqueName: \"kubernetes.io/projected/0c8616f5-2fcb-44c0-b493-ceee84a09e1f-kube-api-access-rq5rf\") pod \"test-curl-7773-9d53ec\" (UID: \"0c8616f5-2fcb-44c0-b493-ceee84a09e1f\") " pod="opendatahub/test-curl-7773-9d53ec" Sep 11 14:33:48.797558 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:48.797536 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq5rf\" (UniqueName: \"kubernetes.io/projected/0c8616f5-2fcb-44c0-b493-ceee84a09e1f-kube-api-access-rq5rf\") pod \"test-curl-7773-9d53ec\" (UID: \"0c8616f5-2fcb-44c0-b493-ceee84a09e1f\") " pod="opendatahub/test-curl-7773-9d53ec" Sep 11 14:33:48.974944 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:48.974916 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-9d53ec" Sep 11 14:33:49.252804 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:49.252776 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7773-9d53ec"] Sep 11 14:33:49.254227 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:33:49.254199 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c8616f5_2fcb_44c0_b493_ceee84a09e1f.slice/crio-746f48effae71e1854ddba89c5dadde51a69c44a9eb46af88e12ad2ae17aca4e WatchSource:0}: Error finding container 746f48effae71e1854ddba89c5dadde51a69c44a9eb46af88e12ad2ae17aca4e: Status 404 returned error can't find the container with id 746f48effae71e1854ddba89c5dadde51a69c44a9eb46af88e12ad2ae17aca4e Sep 11 14:33:49.709424 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:49.709386 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7773-9d53ec" event={"ID":"0c8616f5-2fcb-44c0-b493-ceee84a09e1f","Type":"ContainerStarted","Data":"9604dbdce095fc4cb1762828a7a3e5393edcdefdbace31422b999109538042f3"} Sep 11 14:33:49.709424 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:49.709431 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7773-9d53ec" event={"ID":"0c8616f5-2fcb-44c0-b493-ceee84a09e1f","Type":"ContainerStarted","Data":"746f48effae71e1854ddba89c5dadde51a69c44a9eb46af88e12ad2ae17aca4e"} Sep 11 14:33:49.709932 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:49.709495 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7773-81898e" podUID="2a074075-b43b-42e5-94f9-c9056506a579" containerName="test-curl-7773-81898e" containerID="cri-o://91928ebd112036fffd13fb5c99eb8db9750edbd60b17592d758ada95de633f64" gracePeriod=2 Sep 11 14:33:49.711941 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:49.711917 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7773-81898e\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" podUID="2a074075-b43b-42e5-94f9-c9056506a579" pod="opendatahub/test-curl-7773-81898e" Sep 11 14:33:49.731155 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:49.731106 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7773-9d53ec" podStartSLOduration=1.7310929480000001 podStartE2EDuration="1.731092948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:49.729418133 +0000 UTC m=+1142.066386325" watchObservedRunningTime="2026-09-11 14:33:49.731092948 +0000 UTC m=+1142.068061178" Sep 11 14:33:49.731933 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:49.731901 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7773-81898e\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" podUID="2a074075-b43b-42e5-94f9-c9056506a579" pod="opendatahub/test-curl-7773-81898e" Sep 11 14:33:50.406385 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:50.406303 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7773-9d53ec"] Sep 11 14:33:50.411842 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:50.411800 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7773-9d53ec"] Sep 11 14:33:50.553380 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:50.553342 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7773-9208fb"] Sep 11 14:33:50.553805 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:50.553787 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c8616f5-2fcb-44c0-b493-ceee84a09e1f" containerName="test-curl-7773-9d53ec" Sep 11 14:33:50.553805 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:50.553805 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c8616f5-2fcb-44c0-b493-ceee84a09e1f" containerName="test-curl-7773-9d53ec" Sep 11 14:33:50.553928 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:50.553906 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c8616f5-2fcb-44c0-b493-ceee84a09e1f" containerName="test-curl-7773-9d53ec" Sep 11 14:33:50.556212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:50.556193 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-9208fb" Sep 11 14:33:50.569812 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:50.569775 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7773-9208fb"] Sep 11 14:33:50.606302 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:50.606266 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnjgv\" (UniqueName: \"kubernetes.io/projected/627ad903-d6fd-4fd8-9e36-dc45cb83b9b7-kube-api-access-fnjgv\") pod \"test-curl-7773-9208fb\" (UID: \"627ad903-d6fd-4fd8-9e36-dc45cb83b9b7\") " pod="opendatahub/test-curl-7773-9208fb" Sep 11 14:33:50.706909 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:50.706779 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnjgv\" (UniqueName: \"kubernetes.io/projected/627ad903-d6fd-4fd8-9e36-dc45cb83b9b7-kube-api-access-fnjgv\") pod \"test-curl-7773-9208fb\" (UID: \"627ad903-d6fd-4fd8-9e36-dc45cb83b9b7\") " pod="opendatahub/test-curl-7773-9208fb" Sep 11 14:33:50.715897 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:50.715871 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7773-9d53ec" container="test-curl-7773-9d53ec" Sep 11 14:33:50.716851 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:50.716809 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnjgv\" (UniqueName: \"kubernetes.io/projected/627ad903-d6fd-4fd8-9e36-dc45cb83b9b7-kube-api-access-fnjgv\") pod \"test-curl-7773-9208fb\" (UID: \"627ad903-d6fd-4fd8-9e36-dc45cb83b9b7\") " pod="opendatahub/test-curl-7773-9208fb" Sep 11 14:33:50.878910 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:50.878875 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-9208fb" Sep 11 14:33:51.211033 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:51.211009 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7773-9208fb"] Sep 11 14:33:51.212475 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:33:51.212448 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod627ad903_d6fd_4fd8_9e36_dc45cb83b9b7.slice/crio-d53b62080a794d2ab93d0ee2c8870259a2a290ae104e337df3219f6c7dc7d38d WatchSource:0}: Error finding container d53b62080a794d2ab93d0ee2c8870259a2a290ae104e337df3219f6c7dc7d38d: Status 404 returned error can't find the container with id d53b62080a794d2ab93d0ee2c8870259a2a290ae104e337df3219f6c7dc7d38d Sep 11 14:33:51.723672 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:51.723635 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7773-9208fb" event={"ID":"627ad903-d6fd-4fd8-9e36-dc45cb83b9b7","Type":"ContainerStarted","Data":"f13045ee978e4bbdfb5f71515c0693dc4e6c4063b0ba22da4a9c9a693cc2f308"} Sep 11 14:33:51.724035 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:51.723691 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7773-9208fb" event={"ID":"627ad903-d6fd-4fd8-9e36-dc45cb83b9b7","Type":"ContainerStarted","Data":"d53b62080a794d2ab93d0ee2c8870259a2a290ae104e337df3219f6c7dc7d38d"} Sep 11 14:33:51.724035 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:51.723738 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7773-9d53ec" podUID="0c8616f5-2fcb-44c0-b493-ceee84a09e1f" containerName="test-curl-7773-9d53ec" containerID="cri-o://9604dbdce095fc4cb1762828a7a3e5393edcdefdbace31422b999109538042f3" gracePeriod=2 Sep 11 14:33:51.742009 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:51.741958 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7773-9208fb" podStartSLOduration=1.741942302 podStartE2EDuration="1.741942302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:51.739683297 +0000 UTC m=+1144.076651491" watchObservedRunningTime="2026-09-11 14:33:51.741942302 +0000 UTC m=+1144.078910493" Sep 11 14:33:51.742153 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:51.742047 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7773-9d53ec\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" podUID="0c8616f5-2fcb-44c0-b493-ceee84a09e1f" pod="opendatahub/test-curl-7773-9d53ec" Sep 11 14:33:51.744267 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:51.744244 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7773-9d53ec\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" podUID="0c8616f5-2fcb-44c0-b493-ceee84a09e1f" pod="opendatahub/test-curl-7773-9d53ec" Sep 11 14:33:51.877406 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:51.877383 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-81898e" Sep 11 14:33:51.880154 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:51.880125 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7773-9d53ec\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" podUID="0c8616f5-2fcb-44c0-b493-ceee84a09e1f" pod="opendatahub/test-curl-7773-9d53ec" Sep 11 14:33:51.921967 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:51.921945 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz87r\" (UniqueName: \"kubernetes.io/projected/2a074075-b43b-42e5-94f9-c9056506a579-kube-api-access-cz87r\") pod \"2a074075-b43b-42e5-94f9-c9056506a579\" (UID: \"2a074075-b43b-42e5-94f9-c9056506a579\") " Sep 11 14:33:51.923870 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:51.923846 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a074075-b43b-42e5-94f9-c9056506a579-kube-api-access-cz87r" (OuterVolumeSpecName: "kube-api-access-cz87r") pod "2a074075-b43b-42e5-94f9-c9056506a579" (UID: "2a074075-b43b-42e5-94f9-c9056506a579"). InnerVolumeSpecName "kube-api-access-cz87r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:52.023464 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.023381 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz87r\" (UniqueName: \"kubernetes.io/projected/2a074075-b43b-42e5-94f9-c9056506a579-kube-api-access-cz87r\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:33:52.187958 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.187924 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a074075-b43b-42e5-94f9-c9056506a579" path="/var/lib/kubelet/pods/2a074075-b43b-42e5-94f9-c9056506a579/volumes" Sep 11 14:33:52.390980 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.390949 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7773-9208fb"] Sep 11 14:33:52.396234 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.396205 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7773-9208fb"] Sep 11 14:33:52.533410 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.533376 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7773-73c351"] Sep 11 14:33:52.533808 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.533795 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="627ad903-d6fd-4fd8-9e36-dc45cb83b9b7" containerName="test-curl-7773-9208fb" Sep 11 14:33:52.533872 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.533810 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="627ad903-d6fd-4fd8-9e36-dc45cb83b9b7" containerName="test-curl-7773-9208fb" Sep 11 14:33:52.533916 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.533896 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="627ad903-d6fd-4fd8-9e36-dc45cb83b9b7" containerName="test-curl-7773-9208fb" Sep 11 14:33:52.536328 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.536312 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-73c351" Sep 11 14:33:52.542375 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.542348 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7773-73c351"] Sep 11 14:33:52.630034 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.630002 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6dp\" (UniqueName: \"kubernetes.io/projected/d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3-kube-api-access-ck6dp\") pod \"test-curl-7773-73c351\" (UID: \"d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3\") " pod="opendatahub/test-curl-7773-73c351" Sep 11 14:33:52.730544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.730457 2573 generic.go:356] "Generic (PLEG): container finished" podID="2a074075-b43b-42e5-94f9-c9056506a579" containerID="91928ebd112036fffd13fb5c99eb8db9750edbd60b17592d758ada95de633f64" exitCode=137 Sep 11 14:33:52.730544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.730507 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-81898e" Sep 11 14:33:52.731006 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.730568 2573 scope.go:117] "RemoveContainer" containerID="91928ebd112036fffd13fb5c99eb8db9750edbd60b17592d758ada95de633f64" Sep 11 14:33:52.731006 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.730663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6dp\" (UniqueName: \"kubernetes.io/projected/d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3-kube-api-access-ck6dp\") pod \"test-curl-7773-73c351\" (UID: \"d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3\") " pod="opendatahub/test-curl-7773-73c351" Sep 11 14:33:52.731006 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:52.730858 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7773-9208fb" container="test-curl-7773-9208fb" Sep 11 14:33:52.742640 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.742612 2573 scope.go:117] "RemoveContainer" containerID="91928ebd112036fffd13fb5c99eb8db9750edbd60b17592d758ada95de633f64" Sep 11 14:33:52.742990 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:52.742971 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91928ebd112036fffd13fb5c99eb8db9750edbd60b17592d758ada95de633f64\": container with ID starting with 91928ebd112036fffd13fb5c99eb8db9750edbd60b17592d758ada95de633f64 not found: ID does not exist" containerID="91928ebd112036fffd13fb5c99eb8db9750edbd60b17592d758ada95de633f64" Sep 11 14:33:52.743078 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.743011 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91928ebd112036fffd13fb5c99eb8db9750edbd60b17592d758ada95de633f64"} err="failed to get container status \"91928ebd112036fffd13fb5c99eb8db9750edbd60b17592d758ada95de633f64\": rpc error: code = NotFound desc = could not find container \"91928ebd112036fffd13fb5c99eb8db9750edbd60b17592d758ada95de633f64\": container with ID starting with 91928ebd112036fffd13fb5c99eb8db9750edbd60b17592d758ada95de633f64 not found: ID does not exist" Sep 11 14:33:52.743534 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.743514 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6dp\" (UniqueName: \"kubernetes.io/projected/d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3-kube-api-access-ck6dp\") pod \"test-curl-7773-73c351\" (UID: \"d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3\") " pod="opendatahub/test-curl-7773-73c351" Sep 11 14:33:52.846914 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:52.846878 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-73c351" Sep 11 14:33:53.119636 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:53.119608 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7773-73c351"] Sep 11 14:33:53.120674 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:33:53.120646 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd115ba8c_eac9_4235_a6b2_ab9a7e5a98d3.slice/crio-cdba49a83cf9e5663cc94d16348826cf64258e4d5f3315a617b0a6b85a181087 WatchSource:0}: Error finding container cdba49a83cf9e5663cc94d16348826cf64258e4d5f3315a617b0a6b85a181087: Status 404 returned error can't find the container with id cdba49a83cf9e5663cc94d16348826cf64258e4d5f3315a617b0a6b85a181087 Sep 11 14:33:53.739290 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:53.739263 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7773-73c351" event={"ID":"d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3","Type":"ContainerStarted","Data":"37bb6dd47c91795086fa680238cb156b823deee99ad6767354ed177274ec7aa8"} Sep 11 14:33:53.739650 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:53.739305 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7773-73c351" event={"ID":"d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3","Type":"ContainerStarted","Data":"cdba49a83cf9e5663cc94d16348826cf64258e4d5f3315a617b0a6b85a181087"} Sep 11 14:33:53.739650 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:53.739371 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7773-9208fb" podUID="627ad903-d6fd-4fd8-9e36-dc45cb83b9b7" containerName="test-curl-7773-9208fb" containerID="cri-o://f13045ee978e4bbdfb5f71515c0693dc4e6c4063b0ba22da4a9c9a693cc2f308" gracePeriod=2 Sep 11 14:33:53.742047 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:53.742025 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7773-9208fb\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" podUID="627ad903-d6fd-4fd8-9e36-dc45cb83b9b7" pod="opendatahub/test-curl-7773-9208fb" Sep 11 14:33:53.758127 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:53.758085 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7773-73c351" podStartSLOduration=1.758070571 podStartE2EDuration="1.758070571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:53.757884197 +0000 UTC m=+1146.094852389" watchObservedRunningTime="2026-09-11 14:33:53.758070571 +0000 UTC m=+1146.095038765" Sep 11 14:33:53.760462 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:53.760435 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7773-9208fb\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" podUID="627ad903-d6fd-4fd8-9e36-dc45cb83b9b7" pod="opendatahub/test-curl-7773-9208fb" Sep 11 14:33:53.895960 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:53.895934 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-9d53ec" Sep 11 14:33:53.898507 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:53.898481 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7773-9208fb\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" podUID="627ad903-d6fd-4fd8-9e36-dc45cb83b9b7" pod="opendatahub/test-curl-7773-9208fb" Sep 11 14:33:53.942314 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:53.942291 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rq5rf\" (UniqueName: \"kubernetes.io/projected/0c8616f5-2fcb-44c0-b493-ceee84a09e1f-kube-api-access-rq5rf\") pod \"0c8616f5-2fcb-44c0-b493-ceee84a09e1f\" (UID: \"0c8616f5-2fcb-44c0-b493-ceee84a09e1f\") " Sep 11 14:33:53.944260 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:53.944236 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8616f5-2fcb-44c0-b493-ceee84a09e1f-kube-api-access-rq5rf" (OuterVolumeSpecName: "kube-api-access-rq5rf") pod "0c8616f5-2fcb-44c0-b493-ceee84a09e1f" (UID: "0c8616f5-2fcb-44c0-b493-ceee84a09e1f"). InnerVolumeSpecName "kube-api-access-rq5rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:54.043541 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:54.043472 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq5rf\" (UniqueName: \"kubernetes.io/projected/0c8616f5-2fcb-44c0-b493-ceee84a09e1f-kube-api-access-rq5rf\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:33:54.187273 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:54.187233 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c8616f5-2fcb-44c0-b493-ceee84a09e1f" path="/var/lib/kubelet/pods/0c8616f5-2fcb-44c0-b493-ceee84a09e1f/volumes" Sep 11 14:33:54.422998 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:54.422960 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7773-73c351"] Sep 11 14:33:54.429712 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:54.429687 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7773-73c351"] Sep 11 14:33:54.747265 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:54.747178 2573 generic.go:356] "Generic (PLEG): container finished" podID="0c8616f5-2fcb-44c0-b493-ceee84a09e1f" containerID="9604dbdce095fc4cb1762828a7a3e5393edcdefdbace31422b999109538042f3" exitCode=137 Sep 11 14:33:54.747265 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:54.747235 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-9d53ec" Sep 11 14:33:54.747734 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:54.747277 2573 scope.go:117] "RemoveContainer" containerID="9604dbdce095fc4cb1762828a7a3e5393edcdefdbace31422b999109538042f3" Sep 11 14:33:54.747734 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:54.747614 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7773-73c351" container="test-curl-7773-73c351" Sep 11 14:33:54.756385 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:54.756349 2573 scope.go:117] "RemoveContainer" containerID="9604dbdce095fc4cb1762828a7a3e5393edcdefdbace31422b999109538042f3" Sep 11 14:33:54.756648 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:54.756628 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9604dbdce095fc4cb1762828a7a3e5393edcdefdbace31422b999109538042f3\": container with ID starting with 9604dbdce095fc4cb1762828a7a3e5393edcdefdbace31422b999109538042f3 not found: ID does not exist" containerID="9604dbdce095fc4cb1762828a7a3e5393edcdefdbace31422b999109538042f3" Sep 11 14:33:54.756706 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:54.756656 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9604dbdce095fc4cb1762828a7a3e5393edcdefdbace31422b999109538042f3"} err="failed to get container status \"9604dbdce095fc4cb1762828a7a3e5393edcdefdbace31422b999109538042f3\": rpc error: code = NotFound desc = could not find container \"9604dbdce095fc4cb1762828a7a3e5393edcdefdbace31422b999109538042f3\": container with ID starting with 9604dbdce095fc4cb1762828a7a3e5393edcdefdbace31422b999109538042f3 not found: ID does not exist" Sep 11 14:33:55.482944 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.482911 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt"] Sep 11 14:33:55.483581 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.483557 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3" containerName="test-curl-7773-73c351" Sep 11 14:33:55.483581 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.483580 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3" containerName="test-curl-7773-73c351" Sep 11 14:33:55.483771 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.483684 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3" containerName="test-curl-7773-73c351" Sep 11 14:33:55.486821 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.486799 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.489465 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.489440 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1c057e5e-openshift-default-dockercfg-bsc5r\"" Sep 11 14:33:55.499268 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.499240 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt"] Sep 11 14:33:55.557564 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.557539 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-envoy\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.557694 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.557569 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdr42\" (UniqueName: \"kubernetes.io/projected/20d6c4c8-dd26-4c3a-b654-0be85f02860d-kube-api-access-tdr42\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.557694 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.557589 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-credential-socket\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.557694 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.557617 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-data\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.557827 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.557713 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-workload-socket\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.557827 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.557812 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-workload-certs\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.557913 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.557873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-token\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.557948 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.557910 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istiod-ca-cert\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.557948 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.557930 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-podinfo\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.658610 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.658576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istiod-ca-cert\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.658610 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.658614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-podinfo\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.658883 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.658642 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-envoy\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.658883 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.658665 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdr42\" (UniqueName: \"kubernetes.io/projected/20d6c4c8-dd26-4c3a-b654-0be85f02860d-kube-api-access-tdr42\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.658883 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.658697 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-credential-socket\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.658883 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.658742 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-data\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.658883 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.658782 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-workload-socket\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.659162 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.658893 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-workload-certs\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.659162 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.658996 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-token\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.659260 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.659212 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-credential-socket\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.659333 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.659308 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-workload-certs\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.659525 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.659497 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-data\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.659625 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.659444 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istiod-ca-cert\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.659625 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.659571 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-workload-socket\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.661008 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.660983 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-envoy\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.661240 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.661224 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-podinfo\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.666926 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.666907 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-token\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.667434 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.667413 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdr42\" (UniqueName: \"kubernetes.io/projected/20d6c4c8-dd26-4c3a-b654-0be85f02860d-kube-api-access-tdr42\") pod \"e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.755624 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.755586 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7773-73c351" podUID="d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3" containerName="test-curl-7773-73c351" containerID="cri-o://37bb6dd47c91795086fa680238cb156b823deee99ad6767354ed177274ec7aa8" gracePeriod=2 Sep 11 14:33:55.757982 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:55.757948 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7773-73c351\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" podUID="d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3" pod="opendatahub/test-curl-7773-73c351" Sep 11 14:33:55.799084 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.799055 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:55.908939 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.908918 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-9208fb" Sep 11 14:33:55.911605 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:55.911576 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7773-73c351\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" podUID="d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3" pod="opendatahub/test-curl-7773-73c351" Sep 11 14:33:55.965005 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.964975 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnjgv\" (UniqueName: \"kubernetes.io/projected/627ad903-d6fd-4fd8-9e36-dc45cb83b9b7-kube-api-access-fnjgv\") pod \"627ad903-d6fd-4fd8-9e36-dc45cb83b9b7\" (UID: \"627ad903-d6fd-4fd8-9e36-dc45cb83b9b7\") " Sep 11 14:33:55.966978 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:55.966957 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627ad903-d6fd-4fd8-9e36-dc45cb83b9b7-kube-api-access-fnjgv" (OuterVolumeSpecName: "kube-api-access-fnjgv") pod "627ad903-d6fd-4fd8-9e36-dc45cb83b9b7" (UID: "627ad903-d6fd-4fd8-9e36-dc45cb83b9b7"). InnerVolumeSpecName "kube-api-access-fnjgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:56.066471 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:56.066407 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnjgv\" (UniqueName: \"kubernetes.io/projected/627ad903-d6fd-4fd8-9e36-dc45cb83b9b7-kube-api-access-fnjgv\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:33:56.083465 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:56.083436 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt"] Sep 11 14:33:56.084332 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:33:56.084305 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d6c4c8_dd26_4c3a_b654_0be85f02860d.slice/crio-11e2fba6a37c6c7a165003b0e590ea3986ca32946393605b4b15616c1dfe70b7 WatchSource:0}: Error finding container 11e2fba6a37c6c7a165003b0e590ea3986ca32946393605b4b15616c1dfe70b7: Status 404 returned error can't find the container with id 11e2fba6a37c6c7a165003b0e590ea3986ca32946393605b4b15616c1dfe70b7 Sep 11 14:33:56.086630 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:56.086603 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:33:56.086686 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:56.086666 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:33:56.086718 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:56.086693 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:33:56.188039 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:56.188004 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="627ad903-d6fd-4fd8-9e36-dc45cb83b9b7" path="/var/lib/kubelet/pods/627ad903-d6fd-4fd8-9e36-dc45cb83b9b7/volumes" Sep 11 14:33:56.763103 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:56.763072 2573 generic.go:356] "Generic (PLEG): container finished" podID="627ad903-d6fd-4fd8-9e36-dc45cb83b9b7" containerID="f13045ee978e4bbdfb5f71515c0693dc4e6c4063b0ba22da4a9c9a693cc2f308" exitCode=137 Sep 11 14:33:56.763501 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:56.763117 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-9208fb" Sep 11 14:33:56.763501 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:56.763177 2573 scope.go:117] "RemoveContainer" containerID="f13045ee978e4bbdfb5f71515c0693dc4e6c4063b0ba22da4a9c9a693cc2f308" Sep 11 14:33:56.765131 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:56.765098 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" event={"ID":"20d6c4c8-dd26-4c3a-b654-0be85f02860d","Type":"ContainerStarted","Data":"01be1658cbafd73ca2ec18286636b14e673551fb4767a27927f57fa27d04134b"} Sep 11 14:33:56.765257 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:56.765150 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" event={"ID":"20d6c4c8-dd26-4c3a-b654-0be85f02860d","Type":"ContainerStarted","Data":"11e2fba6a37c6c7a165003b0e590ea3986ca32946393605b4b15616c1dfe70b7"} Sep 11 14:33:56.772544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:56.772528 2573 scope.go:117] "RemoveContainer" containerID="f13045ee978e4bbdfb5f71515c0693dc4e6c4063b0ba22da4a9c9a693cc2f308" Sep 11 14:33:56.772780 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:56.772763 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13045ee978e4bbdfb5f71515c0693dc4e6c4063b0ba22da4a9c9a693cc2f308\": container with ID starting with f13045ee978e4bbdfb5f71515c0693dc4e6c4063b0ba22da4a9c9a693cc2f308 not found: ID does not exist" containerID="f13045ee978e4bbdfb5f71515c0693dc4e6c4063b0ba22da4a9c9a693cc2f308" Sep 11 14:33:56.772844 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:56.772785 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13045ee978e4bbdfb5f71515c0693dc4e6c4063b0ba22da4a9c9a693cc2f308"} err="failed to get container status \"f13045ee978e4bbdfb5f71515c0693dc4e6c4063b0ba22da4a9c9a693cc2f308\": rpc error: code = NotFound desc = could not find container \"f13045ee978e4bbdfb5f71515c0693dc4e6c4063b0ba22da4a9c9a693cc2f308\": container with ID starting with f13045ee978e4bbdfb5f71515c0693dc4e6c4063b0ba22da4a9c9a693cc2f308 not found: ID does not exist" Sep 11 14:33:56.791000 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:56.790963 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" podStartSLOduration=1.790950474 podStartE2EDuration="1.790950474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:56.789182758 +0000 UTC m=+1149.126150949" watchObservedRunningTime="2026-09-11 14:33:56.790950474 +0000 UTC m=+1149.127918665" Sep 11 14:33:56.799739 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:56.799720 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:57.804544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:57.804526 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:57.945004 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:57.944981 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-73c351" Sep 11 14:33:57.985825 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:57.985801 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck6dp\" (UniqueName: \"kubernetes.io/projected/d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3-kube-api-access-ck6dp\") pod \"d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3\" (UID: \"d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3\") " Sep 11 14:33:57.987931 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:57.987912 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3-kube-api-access-ck6dp" (OuterVolumeSpecName: "kube-api-access-ck6dp") pod "d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3" (UID: "d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3"). InnerVolumeSpecName "kube-api-access-ck6dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:58.087050 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:58.087018 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck6dp\" (UniqueName: \"kubernetes.io/projected/d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3-kube-api-access-ck6dp\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:33:58.188711 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:58.188682 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3" path="/var/lib/kubelet/pods/d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3/volumes" Sep 11 14:33:58.781060 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:58.781027 2573 generic.go:356] "Generic (PLEG): container finished" podID="d115ba8c-eac9-4235-a6b2-ab9a7e5a98d3" containerID="37bb6dd47c91795086fa680238cb156b823deee99ad6767354ed177274ec7aa8" exitCode=137 Sep 11 14:33:58.781244 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:58.781080 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7773-73c351" Sep 11 14:33:58.781244 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:58.781127 2573 scope.go:117] "RemoveContainer" containerID="37bb6dd47c91795086fa680238cb156b823deee99ad6767354ed177274ec7aa8" Sep 11 14:33:58.781560 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:58.781534 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:58.783109 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:58.783091 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:33:58.790935 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:58.790916 2573 scope.go:117] "RemoveContainer" containerID="37bb6dd47c91795086fa680238cb156b823deee99ad6767354ed177274ec7aa8" Sep 11 14:33:58.791210 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:33:58.791191 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37bb6dd47c91795086fa680238cb156b823deee99ad6767354ed177274ec7aa8\": container with ID starting with 37bb6dd47c91795086fa680238cb156b823deee99ad6767354ed177274ec7aa8 not found: ID does not exist" containerID="37bb6dd47c91795086fa680238cb156b823deee99ad6767354ed177274ec7aa8" Sep 11 14:33:58.791260 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:33:58.791217 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bb6dd47c91795086fa680238cb156b823deee99ad6767354ed177274ec7aa8"} err="failed to get container status \"37bb6dd47c91795086fa680238cb156b823deee99ad6767354ed177274ec7aa8\": rpc error: code = NotFound desc = could not find container \"37bb6dd47c91795086fa680238cb156b823deee99ad6767354ed177274ec7aa8\": container with ID starting with 37bb6dd47c91795086fa680238cb156b823deee99ad6767354ed177274ec7aa8 not found: ID does not exist" Sep 11 14:34:20.923683 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:20.923647 2573 generic.go:356] "Generic (PLEG): container finished" podID="20d6c4c8-dd26-4c3a-b654-0be85f02860d" containerID="01be1658cbafd73ca2ec18286636b14e673551fb4767a27927f57fa27d04134b" exitCode=0 Sep 11 14:34:20.924243 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:20.923724 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" event={"ID":"20d6c4c8-dd26-4c3a-b654-0be85f02860d","Type":"ContainerDied","Data":"01be1658cbafd73ca2ec18286636b14e673551fb4767a27927f57fa27d04134b"} Sep 11 14:34:20.924318 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:20.924270 2573 scope.go:117] "RemoveContainer" containerID="01be1658cbafd73ca2ec18286636b14e673551fb4767a27927f57fa27d04134b" Sep 11 14:34:20.925413 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:20.925386 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:34:20.925483 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:20.925454 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:34:20.925521 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:20.925483 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:34:21.938622 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:21.938588 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" event={"ID":"20d6c4c8-dd26-4c3a-b654-0be85f02860d","Type":"ContainerStarted","Data":"c893fd31219980cd6256c2831c7019393e3e9fa06a8e5e6de6bf91b82ab25064"} Sep 11 14:34:22.800010 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:22.799968 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:34:22.800205 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:22.800153 2573 patch_prober.go:28] interesting pod/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 11 14:34:22.800254 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:22.800200 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" podUID="20d6c4c8-dd26-4c3a-b654-0be85f02860d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 11 14:34:23.799672 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:23.799638 2573 patch_prober.go:28] interesting pod/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 11 14:34:23.800073 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:23.799693 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" podUID="20d6c4c8-dd26-4c3a-b654-0be85f02860d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 11 14:34:24.799858 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:24.799799 2573 patch_prober.go:28] interesting pod/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 11 14:34:24.800252 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:24.799878 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" podUID="20d6c4c8-dd26-4c3a-b654-0be85f02860d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 11 14:34:25.800103 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:25.800068 2573 patch_prober.go:28] interesting pod/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 11 14:34:25.800481 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:25.800129 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" podUID="20d6c4c8-dd26-4c3a-b654-0be85f02860d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 11 14:34:26.799743 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:26.799651 2573 patch_prober.go:28] interesting pod/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 11 14:34:26.799743 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:26.799710 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" podUID="20d6c4c8-dd26-4c3a-b654-0be85f02860d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 11 14:34:27.799485 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:27.799451 2573 patch_prober.go:28] interesting pod/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 11 14:34:27.799917 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:27.799506 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" podUID="20d6c4c8-dd26-4c3a-b654-0be85f02860d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 11 14:34:28.782238 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:28.782205 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:34:28.804530 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:28.804499 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:34:28.987639 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:34:28.987610 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:35:14.536902 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:14.536846 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v"] Sep 11 14:35:14.542010 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:14.541986 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v" Sep 11 14:35:14.544618 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:14.544598 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:35:14.544743 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:14.544614 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp8gp\"" Sep 11 14:35:14.549896 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:14.549871 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v"] Sep 11 14:35:14.636321 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:14.636291 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c22bc368-4b57-43bc-b361-4e46d4d0ff0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v\" (UID: \"c22bc368-4b57-43bc-b361-4e46d4d0ff0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v" Sep 11 14:35:14.737791 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:14.737761 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c22bc368-4b57-43bc-b361-4e46d4d0ff0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v\" (UID: \"c22bc368-4b57-43bc-b361-4e46d4d0ff0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v" Sep 11 14:35:14.738379 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:14.738357 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c22bc368-4b57-43bc-b361-4e46d4d0ff0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v\" (UID: \"c22bc368-4b57-43bc-b361-4e46d4d0ff0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v" Sep 11 14:35:14.854214 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:14.854183 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v" Sep 11 14:35:15.130346 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:15.130315 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v"] Sep 11 14:35:15.131578 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:35:15.131547 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22bc368_4b57_43bc_b361_4e46d4d0ff0a.slice/crio-194b64b2eaaa243b18d87138d06a9cc239098fc720e5d0af23126b8d4dff2813 WatchSource:0}: Error finding container 194b64b2eaaa243b18d87138d06a9cc239098fc720e5d0af23126b8d4dff2813: Status 404 returned error can't find the container with id 194b64b2eaaa243b18d87138d06a9cc239098fc720e5d0af23126b8d4dff2813 Sep 11 14:35:15.287756 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:15.287719 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v" event={"ID":"c22bc368-4b57-43bc-b361-4e46d4d0ff0a","Type":"ContainerStarted","Data":"194b64b2eaaa243b18d87138d06a9cc239098fc720e5d0af23126b8d4dff2813"} Sep 11 14:35:16.295561 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:16.295521 2573 generic.go:356] "Generic (PLEG): container finished" podID="c22bc368-4b57-43bc-b361-4e46d4d0ff0a" containerID="bba1cdd7f990e1aa7b5614e7deb531480a6c52e0fdd3be3fffa5ec746cb24bcd" exitCode=0 Sep 11 14:35:16.295938 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:16.295607 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v" event={"ID":"c22bc368-4b57-43bc-b361-4e46d4d0ff0a","Type":"ContainerDied","Data":"bba1cdd7f990e1aa7b5614e7deb531480a6c52e0fdd3be3fffa5ec746cb24bcd"} Sep 11 14:35:17.434766 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:17.434742 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v" Sep 11 14:35:17.467165 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:17.467139 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c22bc368-4b57-43bc-b361-4e46d4d0ff0a-maas-api-service-ca\") pod \"c22bc368-4b57-43bc-b361-4e46d4d0ff0a\" (UID: \"c22bc368-4b57-43bc-b361-4e46d4d0ff0a\") " Sep 11 14:35:17.467454 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:17.467431 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22bc368-4b57-43bc-b361-4e46d4d0ff0a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c22bc368-4b57-43bc-b361-4e46d4d0ff0a" (UID: "c22bc368-4b57-43bc-b361-4e46d4d0ff0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:17.568376 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:17.568287 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c22bc368-4b57-43bc-b361-4e46d4d0ff0a-maas-api-service-ca\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:35:18.309565 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:18.309538 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v" Sep 11 14:35:18.309565 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:18.309550 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v" event={"ID":"c22bc368-4b57-43bc-b361-4e46d4d0ff0a","Type":"ContainerDied","Data":"194b64b2eaaa243b18d87138d06a9cc239098fc720e5d0af23126b8d4dff2813"} Sep 11 14:35:18.309757 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:18.309592 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="194b64b2eaaa243b18d87138d06a9cc239098fc720e5d0af23126b8d4dff2813" Sep 11 14:35:30.626078 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:30.626043 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v"] Sep 11 14:35:30.628682 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:30.628662 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1c057e5e-8pb5v"] Sep 11 14:35:32.187547 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:32.187515 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c22bc368-4b57-43bc-b361-4e46d4d0ff0a" path="/var/lib/kubelet/pods/c22bc368-4b57-43bc-b361-4e46d4d0ff0a/volumes" Sep 11 14:35:48.361841 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:48.361761 2573 scope.go:117] "RemoveContainer" containerID="258aa0e3ee6de73003a2777a6d1d61bac387d940847b4c879896489eb4dccf52" Sep 11 14:35:48.370663 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:35:48.370639 2573 scope.go:117] "RemoveContainer" containerID="38683571770ac615d8785b70f78a96b48c1416eba95f2835679b1cffbe436bbd" Sep 11 14:36:03.290070 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:03.290031 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7776-1f547a"] Sep 11 14:36:03.290537 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:03.290473 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22bc368-4b57-43bc-b361-4e46d4d0ff0a" containerName="revoke-keys" Sep 11 14:36:03.290537 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:03.290488 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22bc368-4b57-43bc-b361-4e46d4d0ff0a" containerName="revoke-keys" Sep 11 14:36:03.290614 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:03.290556 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22bc368-4b57-43bc-b361-4e46d4d0ff0a" containerName="revoke-keys" Sep 11 14:36:03.295047 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:03.295030 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-1f547a" Sep 11 14:36:03.298060 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:03.298039 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-998c2\"" Sep 11 14:36:03.303792 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:03.303768 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7776-1f547a"] Sep 11 14:36:03.421846 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:03.421802 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzj5k\" (UniqueName: \"kubernetes.io/projected/4e636581-a448-480c-bd15-b9ae6783f928-kube-api-access-mzj5k\") pod \"test-curl-7776-1f547a\" (UID: \"4e636581-a448-480c-bd15-b9ae6783f928\") " pod="opendatahub/test-curl-7776-1f547a" Sep 11 14:36:03.522948 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:03.522918 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzj5k\" (UniqueName: \"kubernetes.io/projected/4e636581-a448-480c-bd15-b9ae6783f928-kube-api-access-mzj5k\") pod \"test-curl-7776-1f547a\" (UID: \"4e636581-a448-480c-bd15-b9ae6783f928\") " pod="opendatahub/test-curl-7776-1f547a" Sep 11 14:36:03.532107 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:03.532083 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzj5k\" (UniqueName: \"kubernetes.io/projected/4e636581-a448-480c-bd15-b9ae6783f928-kube-api-access-mzj5k\") pod \"test-curl-7776-1f547a\" (UID: \"4e636581-a448-480c-bd15-b9ae6783f928\") " pod="opendatahub/test-curl-7776-1f547a" Sep 11 14:36:03.605283 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:03.605260 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-1f547a" Sep 11 14:36:03.876274 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:03.876246 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7776-1f547a"] Sep 11 14:36:03.878205 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:36:03.878159 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e636581_a448_480c_bd15_b9ae6783f928.slice/crio-259974e51c482ff0e98f8f3b0e12910eb04688bdc22c360fcc4d0416baed15f1 WatchSource:0}: Error finding container 259974e51c482ff0e98f8f3b0e12910eb04688bdc22c360fcc4d0416baed15f1: Status 404 returned error can't find the container with id 259974e51c482ff0e98f8f3b0e12910eb04688bdc22c360fcc4d0416baed15f1 Sep 11 14:36:04.586109 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:04.586072 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7776-1f547a" event={"ID":"4e636581-a448-480c-bd15-b9ae6783f928","Type":"ContainerStarted","Data":"a3358d36ceca44fe54273060565500e4f7ad550f9668fe7a41e5bbd9498083fe"} Sep 11 14:36:04.586506 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:04.586120 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7776-1f547a" event={"ID":"4e636581-a448-480c-bd15-b9ae6783f928","Type":"ContainerStarted","Data":"259974e51c482ff0e98f8f3b0e12910eb04688bdc22c360fcc4d0416baed15f1"} Sep 11 14:36:04.604777 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:04.604729 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7776-1f547a" podStartSLOduration=1.604713613 podStartE2EDuration="1.604713613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:36:04.604502579 +0000 UTC m=+1276.941470771" watchObservedRunningTime="2026-09-11 14:36:04.604713613 +0000 UTC m=+1276.941681803" Sep 11 14:36:05.289983 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:05.289946 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7776-1f547a"] Sep 11 14:36:05.296315 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:05.296285 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7776-1f547a"] Sep 11 14:36:05.592363 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:36:05.592324 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7776-1f547a" container="test-curl-7776-1f547a" Sep 11 14:36:06.598461 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:06.598423 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7776-1f547a" podUID="4e636581-a448-480c-bd15-b9ae6783f928" containerName="test-curl-7776-1f547a" containerID="cri-o://a3358d36ceca44fe54273060565500e4f7ad550f9668fe7a41e5bbd9498083fe" gracePeriod=2 Sep 11 14:36:08.785128 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:08.785106 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-1f547a" Sep 11 14:36:08.878433 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:08.878349 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzj5k\" (UniqueName: \"kubernetes.io/projected/4e636581-a448-480c-bd15-b9ae6783f928-kube-api-access-mzj5k\") pod \"4e636581-a448-480c-bd15-b9ae6783f928\" (UID: \"4e636581-a448-480c-bd15-b9ae6783f928\") " Sep 11 14:36:08.880474 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:08.880441 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e636581-a448-480c-bd15-b9ae6783f928-kube-api-access-mzj5k" (OuterVolumeSpecName: "kube-api-access-mzj5k") pod "4e636581-a448-480c-bd15-b9ae6783f928" (UID: "4e636581-a448-480c-bd15-b9ae6783f928"). InnerVolumeSpecName "kube-api-access-mzj5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:36:08.979438 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:08.979411 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzj5k\" (UniqueName: \"kubernetes.io/projected/4e636581-a448-480c-bd15-b9ae6783f928-kube-api-access-mzj5k\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:36:09.616630 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:09.616593 2573 generic.go:356] "Generic (PLEG): container finished" podID="4e636581-a448-480c-bd15-b9ae6783f928" containerID="a3358d36ceca44fe54273060565500e4f7ad550f9668fe7a41e5bbd9498083fe" exitCode=137 Sep 11 14:36:09.616809 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:09.616642 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-1f547a" Sep 11 14:36:09.616809 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:09.616651 2573 scope.go:117] "RemoveContainer" containerID="a3358d36ceca44fe54273060565500e4f7ad550f9668fe7a41e5bbd9498083fe" Sep 11 14:36:09.626492 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:09.626475 2573 scope.go:117] "RemoveContainer" containerID="a3358d36ceca44fe54273060565500e4f7ad550f9668fe7a41e5bbd9498083fe" Sep 11 14:36:09.626782 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:36:09.626764 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3358d36ceca44fe54273060565500e4f7ad550f9668fe7a41e5bbd9498083fe\": container with ID starting with a3358d36ceca44fe54273060565500e4f7ad550f9668fe7a41e5bbd9498083fe not found: ID does not exist" containerID="a3358d36ceca44fe54273060565500e4f7ad550f9668fe7a41e5bbd9498083fe" Sep 11 14:36:09.626853 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:09.626790 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3358d36ceca44fe54273060565500e4f7ad550f9668fe7a41e5bbd9498083fe"} err="failed to get container status \"a3358d36ceca44fe54273060565500e4f7ad550f9668fe7a41e5bbd9498083fe\": rpc error: code = NotFound desc = could not find container \"a3358d36ceca44fe54273060565500e4f7ad550f9668fe7a41e5bbd9498083fe\": container with ID starting with a3358d36ceca44fe54273060565500e4f7ad550f9668fe7a41e5bbd9498083fe not found: ID does not exist" Sep 11 14:36:09.627971 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:36:09.627951 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7776-1f547a\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" podUID="4e636581-a448-480c-bd15-b9ae6783f928" pod="opendatahub/test-curl-7776-1f547a" Sep 11 14:36:10.187935 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:10.187901 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e636581-a448-480c-bd15-b9ae6783f928" path="/var/lib/kubelet/pods/4e636581-a448-480c-bd15-b9ae6783f928/volumes" Sep 11 14:36:23.593732 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:23.593700 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt"] Sep 11 14:36:23.594156 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:23.593924 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" podUID="20d6c4c8-dd26-4c3a-b654-0be85f02860d" containerName="istio-proxy" containerID="cri-o://c893fd31219980cd6256c2831c7019393e3e9fa06a8e5e6de6bf91b82ab25064" gracePeriod=30 Sep 11 14:36:28.746514 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.746471 2573 generic.go:356] "Generic (PLEG): container finished" podID="20d6c4c8-dd26-4c3a-b654-0be85f02860d" containerID="c893fd31219980cd6256c2831c7019393e3e9fa06a8e5e6de6bf91b82ab25064" exitCode=0 Sep 11 14:36:28.746514 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.746507 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" event={"ID":"20d6c4c8-dd26-4c3a-b654-0be85f02860d","Type":"ContainerDied","Data":"c893fd31219980cd6256c2831c7019393e3e9fa06a8e5e6de6bf91b82ab25064"} Sep 11 14:36:28.747117 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.746592 2573 scope.go:117] "RemoveContainer" containerID="01be1658cbafd73ca2ec18286636b14e673551fb4767a27927f57fa27d04134b" Sep 11 14:36:28.863999 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.863926 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:36:28.972050 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.972012 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-envoy\") pod \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " Sep 11 14:36:28.972252 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.972142 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istiod-ca-cert\") pod \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " Sep 11 14:36:28.972252 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.972186 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-data\") pod \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " Sep 11 14:36:28.972252 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.972239 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-credential-socket\") pod \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " Sep 11 14:36:28.972437 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.972267 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdr42\" (UniqueName: \"kubernetes.io/projected/20d6c4c8-dd26-4c3a-b654-0be85f02860d-kube-api-access-tdr42\") pod \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " Sep 11 14:36:28.972437 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.972298 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-podinfo\") pod \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " Sep 11 14:36:28.972437 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.972340 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-workload-certs\") pod \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " Sep 11 14:36:28.972557 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.972452 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-workload-socket\") pod \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " Sep 11 14:36:28.972557 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.972480 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-token\") pod \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\" (UID: \"20d6c4c8-dd26-4c3a-b654-0be85f02860d\") " Sep 11 14:36:28.972637 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.972605 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "20d6c4c8-dd26-4c3a-b654-0be85f02860d" (UID: "20d6c4c8-dd26-4c3a-b654-0be85f02860d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:36:28.972987 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.972960 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:36:28.973128 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.973091 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "20d6c4c8-dd26-4c3a-b654-0be85f02860d" (UID: "20d6c4c8-dd26-4c3a-b654-0be85f02860d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:36:28.973279 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.973259 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-data" (OuterVolumeSpecName: "istio-data") pod "20d6c4c8-dd26-4c3a-b654-0be85f02860d" (UID: "20d6c4c8-dd26-4c3a-b654-0be85f02860d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:36:28.973279 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.973268 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "20d6c4c8-dd26-4c3a-b654-0be85f02860d" (UID: "20d6c4c8-dd26-4c3a-b654-0be85f02860d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:36:28.973413 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.973397 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "20d6c4c8-dd26-4c3a-b654-0be85f02860d" (UID: "20d6c4c8-dd26-4c3a-b654-0be85f02860d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:36:28.974862 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.974809 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "20d6c4c8-dd26-4c3a-b654-0be85f02860d" (UID: "20d6c4c8-dd26-4c3a-b654-0be85f02860d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:36:28.974971 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.974887 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "20d6c4c8-dd26-4c3a-b654-0be85f02860d" (UID: "20d6c4c8-dd26-4c3a-b654-0be85f02860d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:36:28.975168 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.975141 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-token" (OuterVolumeSpecName: "istio-token") pod "20d6c4c8-dd26-4c3a-b654-0be85f02860d" (UID: "20d6c4c8-dd26-4c3a-b654-0be85f02860d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:36:28.975274 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:28.975208 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d6c4c8-dd26-4c3a-b654-0be85f02860d-kube-api-access-tdr42" (OuterVolumeSpecName: "kube-api-access-tdr42") pod "20d6c4c8-dd26-4c3a-b654-0be85f02860d" (UID: "20d6c4c8-dd26-4c3a-b654-0be85f02860d"). InnerVolumeSpecName "kube-api-access-tdr42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:36:29.074139 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:29.074102 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:36:29.074139 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:29.074133 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:36:29.074139 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:29.074143 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:36:29.074359 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:29.074153 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:36:29.074359 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:29.074162 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:36:29.074359 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:29.074170 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:36:29.074359 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:29.074180 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdr42\" (UniqueName: \"kubernetes.io/projected/20d6c4c8-dd26-4c3a-b654-0be85f02860d-kube-api-access-tdr42\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:36:29.074359 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:29.074187 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d6c4c8-dd26-4c3a-b654-0be85f02860d-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:36:29.754923 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:29.754897 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" Sep 11 14:36:29.755345 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:29.754957 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt" event={"ID":"20d6c4c8-dd26-4c3a-b654-0be85f02860d","Type":"ContainerDied","Data":"11e2fba6a37c6c7a165003b0e590ea3986ca32946393605b4b15616c1dfe70b7"} Sep 11 14:36:29.755345 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:29.755027 2573 scope.go:117] "RemoveContainer" containerID="c893fd31219980cd6256c2831c7019393e3e9fa06a8e5e6de6bf91b82ab25064" Sep 11 14:36:29.779052 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:29.779031 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt"] Sep 11 14:36:29.786273 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:29.786253 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1c057e5e-openshift-default-558d996648-qtxbt"] Sep 11 14:36:30.191708 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:30.191667 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20d6c4c8-dd26-4c3a-b654-0be85f02860d" path="/var/lib/kubelet/pods/20d6c4c8-dd26-4c3a-b654-0be85f02860d/volumes" Sep 11 14:36:48.398438 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:48.398403 2573 scope.go:117] "RemoveContainer" containerID="c7ccd16a24b817166f89d742a7cbb532f2fdb8e60ec60313ff91001e427e40e2" Sep 11 14:36:57.919790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:57.919703 2573 generic.go:356] "Generic (PLEG): container finished" podID="ff9827c8-e340-4b45-9b2f-45c3d53932ad" containerID="9c66ebeb7a8897ee280a5e8748366ab9cedabbb4c75c9cca78cf40a11892d830" exitCode=1 Sep 11 14:36:57.919790 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:57.919780 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" event={"ID":"ff9827c8-e340-4b45-9b2f-45c3d53932ad","Type":"ContainerDied","Data":"9c66ebeb7a8897ee280a5e8748366ab9cedabbb4c75c9cca78cf40a11892d830"} Sep 11 14:36:57.920289 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:57.920184 2573 scope.go:117] "RemoveContainer" containerID="9c66ebeb7a8897ee280a5e8748366ab9cedabbb4c75c9cca78cf40a11892d830" Sep 11 14:36:57.920538 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:57.920523 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:36:57.921560 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:57.921543 2573 generic.go:356] "Generic (PLEG): container finished" podID="520952ba-90cd-4bb1-818f-a5aeb4c54de4" containerID="f69b867e5d58ac2305258666897acfaed25a7c524ed559b0154c74e826081464" exitCode=1 Sep 11 14:36:57.921652 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:57.921598 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" event={"ID":"520952ba-90cd-4bb1-818f-a5aeb4c54de4","Type":"ContainerDied","Data":"f69b867e5d58ac2305258666897acfaed25a7c524ed559b0154c74e826081464"} Sep 11 14:36:57.921950 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:57.921933 2573 scope.go:117] "RemoveContainer" containerID="f69b867e5d58ac2305258666897acfaed25a7c524ed559b0154c74e826081464" Sep 11 14:36:58.766333 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:58.766254 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" Sep 11 14:36:58.929911 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:58.929876 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" event={"ID":"ff9827c8-e340-4b45-9b2f-45c3d53932ad","Type":"ContainerStarted","Data":"86f4999e8a410b14dde58d80992ff7afe82405a2f3775b84c3ae82fc471f740b"} Sep 11 14:36:58.930351 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:58.929946 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" Sep 11 14:36:58.931845 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:58.931813 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" event={"ID":"520952ba-90cd-4bb1-818f-a5aeb4c54de4","Type":"ContainerStarted","Data":"5d827fff868241835477bc1a2440eb33cfab893bdb3c0869a5bab1f4a8c095cd"} Sep 11 14:36:58.932026 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:36:58.932010 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:37:09.939982 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:09.939947 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c4cd778db-mjktf" Sep 11 14:37:09.940368 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:09.940224 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-g9xn5" Sep 11 14:37:48.644476 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.644441 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b"] Sep 11 14:37:48.645092 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.645076 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20d6c4c8-dd26-4c3a-b654-0be85f02860d" containerName="istio-proxy" Sep 11 14:37:48.645149 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.645095 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="20d6c4c8-dd26-4c3a-b654-0be85f02860d" containerName="istio-proxy" Sep 11 14:37:48.645149 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.645136 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20d6c4c8-dd26-4c3a-b654-0be85f02860d" containerName="istio-proxy" Sep 11 14:37:48.645149 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.645145 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="20d6c4c8-dd26-4c3a-b654-0be85f02860d" containerName="istio-proxy" Sep 11 14:37:48.645244 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.645160 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e636581-a448-480c-bd15-b9ae6783f928" containerName="test-curl-7776-1f547a" Sep 11 14:37:48.645244 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.645170 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e636581-a448-480c-bd15-b9ae6783f928" containerName="test-curl-7776-1f547a" Sep 11 14:37:48.645308 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.645256 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="20d6c4c8-dd26-4c3a-b654-0be85f02860d" containerName="istio-proxy" Sep 11 14:37:48.645308 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.645265 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="20d6c4c8-dd26-4c3a-b654-0be85f02860d" containerName="istio-proxy" Sep 11 14:37:48.645308 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.645282 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e636581-a448-480c-bd15-b9ae6783f928" containerName="test-curl-7776-1f547a" Sep 11 14:37:48.648562 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.648539 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.656237 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.656215 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2cabda-openshift-default-dockercfg-57cbq\"" Sep 11 14:37:48.668002 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.667971 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b"] Sep 11 14:37:48.744168 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.744127 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-workload-certs\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.744438 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.744412 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ef2d79-ce56-416e-885c-bf5591f1082e-istiod-ca-cert\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.744577 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.744561 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-podinfo\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.744721 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.744707 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-token\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.744859 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.744827 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-workload-socket\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.745020 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.745003 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zgcr\" (UniqueName: \"kubernetes.io/projected/80ef2d79-ce56-416e-885c-bf5591f1082e-kube-api-access-2zgcr\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.745168 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.745155 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-credential-socket\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.745302 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.745288 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-data\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.745400 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.745388 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-envoy\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.846248 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.846212 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-workload-certs\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.846428 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.846258 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ef2d79-ce56-416e-885c-bf5591f1082e-istiod-ca-cert\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.846428 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.846296 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-podinfo\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.846428 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.846343 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-token\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.846428 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.846376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-workload-socket\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.846428 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.846425 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zgcr\" (UniqueName: \"kubernetes.io/projected/80ef2d79-ce56-416e-885c-bf5591f1082e-kube-api-access-2zgcr\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.846738 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.846481 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-credential-socket\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.846738 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.846531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-data\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.846738 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.846556 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-envoy\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.846738 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.846593 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-workload-certs\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.847134 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.847078 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ef2d79-ce56-416e-885c-bf5591f1082e-istiod-ca-cert\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.847228 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.847189 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-data\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.847228 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.847209 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-credential-socket\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.847343 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.847242 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-workload-socket\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.848812 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.848794 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-envoy\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.849209 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.849191 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-podinfo\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.859909 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.859890 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-token\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.860504 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.860483 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zgcr\" (UniqueName: \"kubernetes.io/projected/80ef2d79-ce56-416e-885c-bf5591f1082e-kube-api-access-2zgcr\") pod \"e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:48.959461 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:48.959397 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:49.263179 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:49.263152 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b"] Sep 11 14:37:49.264540 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:37:49.264515 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ef2d79_ce56_416e_885c_bf5591f1082e.slice/crio-de473a765c992df247a1e5784508abe450953f04c74444ee1691a90385414adf WatchSource:0}: Error finding container de473a765c992df247a1e5784508abe450953f04c74444ee1691a90385414adf: Status 404 returned error can't find the container with id de473a765c992df247a1e5784508abe450953f04c74444ee1691a90385414adf Sep 11 14:37:49.266678 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:49.266644 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:37:49.266776 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:49.266720 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:37:49.266776 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:49.266762 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:37:50.252301 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:50.252266 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" event={"ID":"80ef2d79-ce56-416e-885c-bf5591f1082e","Type":"ContainerStarted","Data":"9537c6fbdb3035931259ef2b6b1ce1ee3a9bd6ff0501fbd766e39fd93ad06424"} Sep 11 14:37:50.252664 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:50.252320 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" event={"ID":"80ef2d79-ce56-416e-885c-bf5591f1082e","Type":"ContainerStarted","Data":"de473a765c992df247a1e5784508abe450953f04c74444ee1691a90385414adf"} Sep 11 14:37:50.281533 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:50.281483 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" podStartSLOduration=2.28146885 podStartE2EDuration="2.28146885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:37:50.278012041 +0000 UTC m=+1382.614980232" watchObservedRunningTime="2026-09-11 14:37:50.28146885 +0000 UTC m=+1382.618437041" Sep 11 14:37:50.960263 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:50.960232 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:50.965685 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:50.965660 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:51.258583 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:51.258498 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:37:51.259411 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:37:51.259385 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:38:06.749858 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:06.749071 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7776-49c0ca"] Sep 11 14:38:06.754524 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:06.754496 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-49c0ca" Sep 11 14:38:06.755520 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:06.755495 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7776-49c0ca"] Sep 11 14:38:06.758212 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:06.758184 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-998c2\"" Sep 11 14:38:06.843497 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:06.843459 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8xn\" (UniqueName: \"kubernetes.io/projected/15af9daa-dcdc-4230-9d9f-08a39d235556-kube-api-access-bw8xn\") pod \"test-curl-7776-49c0ca\" (UID: \"15af9daa-dcdc-4230-9d9f-08a39d235556\") " pod="opendatahub/test-curl-7776-49c0ca" Sep 11 14:38:06.944404 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:06.944351 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8xn\" (UniqueName: \"kubernetes.io/projected/15af9daa-dcdc-4230-9d9f-08a39d235556-kube-api-access-bw8xn\") pod \"test-curl-7776-49c0ca\" (UID: \"15af9daa-dcdc-4230-9d9f-08a39d235556\") " pod="opendatahub/test-curl-7776-49c0ca" Sep 11 14:38:06.959868 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:06.959816 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8xn\" (UniqueName: \"kubernetes.io/projected/15af9daa-dcdc-4230-9d9f-08a39d235556-kube-api-access-bw8xn\") pod \"test-curl-7776-49c0ca\" (UID: \"15af9daa-dcdc-4230-9d9f-08a39d235556\") " pod="opendatahub/test-curl-7776-49c0ca" Sep 11 14:38:07.066008 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:07.065914 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-49c0ca" Sep 11 14:38:07.345241 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:07.342057 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7776-49c0ca"] Sep 11 14:38:07.365067 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:07.364983 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7776-49c0ca" event={"ID":"15af9daa-dcdc-4230-9d9f-08a39d235556","Type":"ContainerStarted","Data":"39b6149bc68861a5fb855a3694a91ce5c629b5a8468836e753ecd93d2242808d"} Sep 11 14:38:08.372949 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:08.372916 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7776-49c0ca" event={"ID":"15af9daa-dcdc-4230-9d9f-08a39d235556","Type":"ContainerStarted","Data":"ee4fd1f2ae51b5ceb9de99098cb0df101d43ad6c33bfb7097ba4aff56680c6a9"} Sep 11 14:38:08.393690 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:08.393639 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7776-49c0ca" podStartSLOduration=2.393624993 podStartE2EDuration="2.393624993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:08.392548089 +0000 UTC m=+1400.729516281" watchObservedRunningTime="2026-09-11 14:38:08.393624993 +0000 UTC m=+1400.730593224" Sep 11 14:38:09.076619 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:09.076524 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7776-49c0ca"] Sep 11 14:38:09.082650 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:09.082619 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7776-49c0ca"] Sep 11 14:38:09.209476 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:09.209440 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7776-1464bb"] Sep 11 14:38:09.209983 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:09.209963 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15af9daa-dcdc-4230-9d9f-08a39d235556" containerName="test-curl-7776-49c0ca" Sep 11 14:38:09.210094 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:09.209984 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="15af9daa-dcdc-4230-9d9f-08a39d235556" containerName="test-curl-7776-49c0ca" Sep 11 14:38:09.210152 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:09.210098 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="15af9daa-dcdc-4230-9d9f-08a39d235556" containerName="test-curl-7776-49c0ca" Sep 11 14:38:09.213271 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:09.213252 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-1464bb" Sep 11 14:38:09.222965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:09.222943 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7776-1464bb"] Sep 11 14:38:09.370423 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:09.370394 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97td4\" (UniqueName: \"kubernetes.io/projected/6ba51da6-7cbf-43a5-9772-249b026ef45e-kube-api-access-97td4\") pod \"test-curl-7776-1464bb\" (UID: \"6ba51da6-7cbf-43a5-9772-249b026ef45e\") " pod="opendatahub/test-curl-7776-1464bb" Sep 11 14:38:09.378525 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:38:09.378504 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7776-49c0ca" container="test-curl-7776-49c0ca" Sep 11 14:38:09.471762 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:09.471728 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97td4\" (UniqueName: \"kubernetes.io/projected/6ba51da6-7cbf-43a5-9772-249b026ef45e-kube-api-access-97td4\") pod \"test-curl-7776-1464bb\" (UID: \"6ba51da6-7cbf-43a5-9772-249b026ef45e\") " pod="opendatahub/test-curl-7776-1464bb" Sep 11 14:38:09.481768 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:09.481742 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97td4\" (UniqueName: \"kubernetes.io/projected/6ba51da6-7cbf-43a5-9772-249b026ef45e-kube-api-access-97td4\") pod \"test-curl-7776-1464bb\" (UID: \"6ba51da6-7cbf-43a5-9772-249b026ef45e\") " pod="opendatahub/test-curl-7776-1464bb" Sep 11 14:38:09.524110 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:09.524085 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-1464bb" Sep 11 14:38:09.842450 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:09.842414 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7776-1464bb"] Sep 11 14:38:09.853950 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:38:09.853903 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ba51da6_7cbf_43a5_9772_249b026ef45e.slice/crio-f08d48fff766799834872b8d400d0120f0ccdc2051203900b316488aa3036990 WatchSource:0}: Error finding container f08d48fff766799834872b8d400d0120f0ccdc2051203900b316488aa3036990: Status 404 returned error can't find the container with id f08d48fff766799834872b8d400d0120f0ccdc2051203900b316488aa3036990 Sep 11 14:38:10.385959 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:10.385924 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7776-1464bb" event={"ID":"6ba51da6-7cbf-43a5-9772-249b026ef45e","Type":"ContainerStarted","Data":"11a8adf04efa129317635d2ed3b34f7e751fb23d8bbfe94c835831bd8b0f8d32"} Sep 11 14:38:10.386368 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:10.385970 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7776-1464bb" event={"ID":"6ba51da6-7cbf-43a5-9772-249b026ef45e","Type":"ContainerStarted","Data":"f08d48fff766799834872b8d400d0120f0ccdc2051203900b316488aa3036990"} Sep 11 14:38:10.386368 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:10.386050 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7776-49c0ca" podUID="15af9daa-dcdc-4230-9d9f-08a39d235556" containerName="test-curl-7776-49c0ca" containerID="cri-o://ee4fd1f2ae51b5ceb9de99098cb0df101d43ad6c33bfb7097ba4aff56680c6a9" gracePeriod=2 Sep 11 14:38:10.428814 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:10.428770 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7776-1464bb" podStartSLOduration=1.428754444 podStartE2EDuration="1.428754444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:10.426678578 +0000 UTC m=+1402.763646770" watchObservedRunningTime="2026-09-11 14:38:10.428754444 +0000 UTC m=+1402.765722634" Sep 11 14:38:11.135419 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:11.135372 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7776-1464bb"] Sep 11 14:38:11.138789 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:11.138758 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7776-1464bb"] Sep 11 14:38:11.391470 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:38:11.391438 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7776-1464bb" container="test-curl-7776-1464bb" Sep 11 14:38:12.399328 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:12.399289 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7776-1464bb" podUID="6ba51da6-7cbf-43a5-9772-249b026ef45e" containerName="test-curl-7776-1464bb" containerID="cri-o://11a8adf04efa129317635d2ed3b34f7e751fb23d8bbfe94c835831bd8b0f8d32" gracePeriod=2 Sep 11 14:38:12.525471 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:12.525446 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-49c0ca" Sep 11 14:38:12.706325 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:12.706250 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw8xn\" (UniqueName: \"kubernetes.io/projected/15af9daa-dcdc-4230-9d9f-08a39d235556-kube-api-access-bw8xn\") pod \"15af9daa-dcdc-4230-9d9f-08a39d235556\" (UID: \"15af9daa-dcdc-4230-9d9f-08a39d235556\") " Sep 11 14:38:12.708497 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:12.708459 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15af9daa-dcdc-4230-9d9f-08a39d235556-kube-api-access-bw8xn" (OuterVolumeSpecName: "kube-api-access-bw8xn") pod "15af9daa-dcdc-4230-9d9f-08a39d235556" (UID: "15af9daa-dcdc-4230-9d9f-08a39d235556"). InnerVolumeSpecName "kube-api-access-bw8xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:38:12.808326 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:12.808292 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw8xn\" (UniqueName: \"kubernetes.io/projected/15af9daa-dcdc-4230-9d9f-08a39d235556-kube-api-access-bw8xn\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:38:13.406800 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:13.406766 2573 generic.go:356] "Generic (PLEG): container finished" podID="80ef2d79-ce56-416e-885c-bf5591f1082e" containerID="9537c6fbdb3035931259ef2b6b1ce1ee3a9bd6ff0501fbd766e39fd93ad06424" exitCode=0 Sep 11 14:38:13.407220 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:13.406867 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" event={"ID":"80ef2d79-ce56-416e-885c-bf5591f1082e","Type":"ContainerDied","Data":"9537c6fbdb3035931259ef2b6b1ce1ee3a9bd6ff0501fbd766e39fd93ad06424"} Sep 11 14:38:13.407438 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:13.407421 2573 scope.go:117] "RemoveContainer" containerID="9537c6fbdb3035931259ef2b6b1ce1ee3a9bd6ff0501fbd766e39fd93ad06424" Sep 11 14:38:13.408277 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:13.408193 2573 generic.go:356] "Generic (PLEG): container finished" podID="15af9daa-dcdc-4230-9d9f-08a39d235556" containerID="ee4fd1f2ae51b5ceb9de99098cb0df101d43ad6c33bfb7097ba4aff56680c6a9" exitCode=137 Sep 11 14:38:13.408277 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:13.408247 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-49c0ca" Sep 11 14:38:13.408277 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:13.408271 2573 scope.go:117] "RemoveContainer" containerID="ee4fd1f2ae51b5ceb9de99098cb0df101d43ad6c33bfb7097ba4aff56680c6a9" Sep 11 14:38:13.408686 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:13.408656 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:38:13.408783 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:13.408726 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:38:13.408783 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:13.408758 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:38:13.421672 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:13.421656 2573 scope.go:117] "RemoveContainer" containerID="ee4fd1f2ae51b5ceb9de99098cb0df101d43ad6c33bfb7097ba4aff56680c6a9" Sep 11 14:38:13.421946 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:38:13.421926 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4fd1f2ae51b5ceb9de99098cb0df101d43ad6c33bfb7097ba4aff56680c6a9\": container with ID starting with ee4fd1f2ae51b5ceb9de99098cb0df101d43ad6c33bfb7097ba4aff56680c6a9 not found: ID does not exist" containerID="ee4fd1f2ae51b5ceb9de99098cb0df101d43ad6c33bfb7097ba4aff56680c6a9" Sep 11 14:38:13.422035 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:13.421950 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4fd1f2ae51b5ceb9de99098cb0df101d43ad6c33bfb7097ba4aff56680c6a9"} err="failed to get container status \"ee4fd1f2ae51b5ceb9de99098cb0df101d43ad6c33bfb7097ba4aff56680c6a9\": rpc error: code = NotFound desc = could not find container \"ee4fd1f2ae51b5ceb9de99098cb0df101d43ad6c33bfb7097ba4aff56680c6a9\": container with ID starting with ee4fd1f2ae51b5ceb9de99098cb0df101d43ad6c33bfb7097ba4aff56680c6a9 not found: ID does not exist" Sep 11 14:38:13.441633 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:38:13.441600 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7776-49c0ca\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" podUID="15af9daa-dcdc-4230-9d9f-08a39d235556" pod="opendatahub/test-curl-7776-49c0ca" Sep 11 14:38:14.191264 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:14.191226 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15af9daa-dcdc-4230-9d9f-08a39d235556" path="/var/lib/kubelet/pods/15af9daa-dcdc-4230-9d9f-08a39d235556/volumes" Sep 11 14:38:14.417022 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:14.416992 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" event={"ID":"80ef2d79-ce56-416e-885c-bf5591f1082e","Type":"ContainerStarted","Data":"e2917142393a5f8a25d781b34fa68bb81488ce0577a71a7922398728c4e48069"} Sep 11 14:38:14.593504 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:14.593481 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-1464bb" Sep 11 14:38:14.738900 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:14.738778 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97td4\" (UniqueName: \"kubernetes.io/projected/6ba51da6-7cbf-43a5-9772-249b026ef45e-kube-api-access-97td4\") pod \"6ba51da6-7cbf-43a5-9772-249b026ef45e\" (UID: \"6ba51da6-7cbf-43a5-9772-249b026ef45e\") " Sep 11 14:38:14.749935 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:14.741572 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba51da6-7cbf-43a5-9772-249b026ef45e-kube-api-access-97td4" (OuterVolumeSpecName: "kube-api-access-97td4") pod "6ba51da6-7cbf-43a5-9772-249b026ef45e" (UID: "6ba51da6-7cbf-43a5-9772-249b026ef45e"). InnerVolumeSpecName "kube-api-access-97td4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:38:14.840804 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:14.840757 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97td4\" (UniqueName: \"kubernetes.io/projected/6ba51da6-7cbf-43a5-9772-249b026ef45e-kube-api-access-97td4\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:38:14.960045 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:14.960004 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:38:14.960224 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:14.960203 2573 patch_prober.go:28] interesting pod/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 14:38:14.960271 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:14.960249 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" podUID="80ef2d79-ce56-416e-885c-bf5591f1082e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 14:38:15.426178 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:15.426132 2573 generic.go:356] "Generic (PLEG): container finished" podID="6ba51da6-7cbf-43a5-9772-249b026ef45e" containerID="11a8adf04efa129317635d2ed3b34f7e751fb23d8bbfe94c835831bd8b0f8d32" exitCode=137 Sep 11 14:38:15.426668 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:15.426216 2573 scope.go:117] "RemoveContainer" containerID="11a8adf04efa129317635d2ed3b34f7e751fb23d8bbfe94c835831bd8b0f8d32" Sep 11 14:38:15.426668 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:15.426279 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7776-1464bb" Sep 11 14:38:15.447335 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:15.447312 2573 scope.go:117] "RemoveContainer" containerID="11a8adf04efa129317635d2ed3b34f7e751fb23d8bbfe94c835831bd8b0f8d32" Sep 11 14:38:15.448393 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:38:15.448358 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a8adf04efa129317635d2ed3b34f7e751fb23d8bbfe94c835831bd8b0f8d32\": container with ID starting with 11a8adf04efa129317635d2ed3b34f7e751fb23d8bbfe94c835831bd8b0f8d32 not found: ID does not exist" containerID="11a8adf04efa129317635d2ed3b34f7e751fb23d8bbfe94c835831bd8b0f8d32" Sep 11 14:38:15.448493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:15.448402 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a8adf04efa129317635d2ed3b34f7e751fb23d8bbfe94c835831bd8b0f8d32"} err="failed to get container status \"11a8adf04efa129317635d2ed3b34f7e751fb23d8bbfe94c835831bd8b0f8d32\": rpc error: code = NotFound desc = could not find container \"11a8adf04efa129317635d2ed3b34f7e751fb23d8bbfe94c835831bd8b0f8d32\": container with ID starting with 11a8adf04efa129317635d2ed3b34f7e751fb23d8bbfe94c835831bd8b0f8d32 not found: ID does not exist" Sep 11 14:38:15.452154 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:38:15.452119 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7776-1464bb\" is forbidden: User \"system:node:ip-10-0-131-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-110.ec2.internal' and this object" podUID="6ba51da6-7cbf-43a5-9772-249b026ef45e" pod="opendatahub/test-curl-7776-1464bb" Sep 11 14:38:15.960009 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:15.959968 2573 patch_prober.go:28] interesting pod/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 14:38:15.960184 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:15.960035 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" podUID="80ef2d79-ce56-416e-885c-bf5591f1082e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 14:38:16.187722 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:16.187688 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ba51da6-7cbf-43a5-9772-249b026ef45e" path="/var/lib/kubelet/pods/6ba51da6-7cbf-43a5-9772-249b026ef45e/volumes" Sep 11 14:38:16.959895 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:16.959855 2573 patch_prober.go:28] interesting pod/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 14:38:16.960279 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:16.959918 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" podUID="80ef2d79-ce56-416e-885c-bf5591f1082e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 14:38:17.959792 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:17.959753 2573 patch_prober.go:28] interesting pod/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 14:38:17.959988 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:17.959809 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" podUID="80ef2d79-ce56-416e-885c-bf5591f1082e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 14:38:18.960673 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:18.960633 2573 patch_prober.go:28] interesting pod/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 14:38:18.961142 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:18.960695 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" podUID="80ef2d79-ce56-416e-885c-bf5591f1082e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 14:38:19.963808 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:19.963776 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:38:19.964185 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:19.964125 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:38:19.964810 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:19.964790 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:38:24.239576 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.239542 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926"] Sep 11 14:38:24.239987 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.239964 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba51da6-7cbf-43a5-9772-249b026ef45e" containerName="test-curl-7776-1464bb" Sep 11 14:38:24.239987 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.239977 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba51da6-7cbf-43a5-9772-249b026ef45e" containerName="test-curl-7776-1464bb" Sep 11 14:38:24.240088 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.240078 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ba51da6-7cbf-43a5-9772-249b026ef45e" containerName="test-curl-7776-1464bb" Sep 11 14:38:24.242207 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.242190 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.245210 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.245190 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-96e743-openshift-default-dockercfg-vdf7g\"" Sep 11 14:38:24.259766 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.259743 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926"] Sep 11 14:38:24.333521 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.333490 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxkx6\" (UniqueName: \"kubernetes.io/projected/135435c4-a57d-48a7-a631-db139fc0d5eb-kube-api-access-fxkx6\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.333687 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.333539 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-data\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.333687 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.333607 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-credential-socket\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.333687 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.333676 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-workload-socket\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.333867 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.333705 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-envoy\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.333867 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.333757 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-workload-certs\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.333867 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.333780 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-podinfo\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.333867 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.333809 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-token\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.333867 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.333865 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135435c4-a57d-48a7-a631-db139fc0d5eb-istiod-ca-cert\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.434928 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.434900 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxkx6\" (UniqueName: \"kubernetes.io/projected/135435c4-a57d-48a7-a631-db139fc0d5eb-kube-api-access-fxkx6\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.435033 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.434946 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-data\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.435033 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.434978 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-credential-socket\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.435033 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.435018 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-workload-socket\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.435200 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.435044 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-envoy\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.435200 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.435111 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-workload-certs\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.435200 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.435143 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-podinfo\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.435200 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.435169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-token\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.435200 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.435192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135435c4-a57d-48a7-a631-db139fc0d5eb-istiod-ca-cert\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.435487 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.435362 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-data\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.435487 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.435442 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-credential-socket\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.435571 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.435491 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-workload-socket\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.435817 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.435788 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135435c4-a57d-48a7-a631-db139fc0d5eb-istiod-ca-cert\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.435817 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.435802 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-workload-certs\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.437467 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.437450 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-envoy\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.437733 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.437714 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-podinfo\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.451102 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.451074 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-token\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.474256 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.474234 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxkx6\" (UniqueName: \"kubernetes.io/projected/135435c4-a57d-48a7-a631-db139fc0d5eb-kube-api-access-fxkx6\") pod \"e2e-api-b-96e743-openshift-default-5cc55d597d-zb926\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.552092 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.552009 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:24.830520 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.830487 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926"] Sep 11 14:38:24.832359 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:38:24.832332 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135435c4_a57d_48a7_a631_db139fc0d5eb.slice/crio-e1af748d766b8ff012a9c839452d5feb03e5ae05ba28b4f8e081889a054fbe9a WatchSource:0}: Error finding container e1af748d766b8ff012a9c839452d5feb03e5ae05ba28b4f8e081889a054fbe9a: Status 404 returned error can't find the container with id e1af748d766b8ff012a9c839452d5feb03e5ae05ba28b4f8e081889a054fbe9a Sep 11 14:38:24.834562 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.834529 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:38:24.834657 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.834606 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:38:24.834657 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:24.834649 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:38:25.511338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:25.511302 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" event={"ID":"135435c4-a57d-48a7-a631-db139fc0d5eb","Type":"ContainerStarted","Data":"3ca22a64a5e171adeb729b2b4f8b703952746589a5e1a1478924381af9e78225"} Sep 11 14:38:25.511730 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:25.511355 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" event={"ID":"135435c4-a57d-48a7-a631-db139fc0d5eb","Type":"ContainerStarted","Data":"e1af748d766b8ff012a9c839452d5feb03e5ae05ba28b4f8e081889a054fbe9a"} Sep 11 14:38:25.553054 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:25.553030 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:25.557954 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:25.557931 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:26.517811 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:26.517785 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:26.518702 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:26.518682 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:38:28.449949 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:38:28.449898 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" podStartSLOduration=4.449881293 podStartE2EDuration="4.449881293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:28.440798303 +0000 UTC m=+1420.777766493" watchObservedRunningTime="2026-09-11 14:38:28.449881293 +0000 UTC m=+1420.786849484" Sep 11 14:39:12.508689 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:12.508644 2573 patch_prober.go:28] interesting pod/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": read tcp 10.132.0.2:56790->10.132.0.81:15021: read: connection reset by peer" start-of-body= Sep 11 14:39:12.509121 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:12.508723 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" podUID="135435c4-a57d-48a7-a631-db139fc0d5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": read tcp 10.132.0.2:56790->10.132.0.81:15021: read: connection reset by peer" Sep 11 14:39:12.803051 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:12.802960 2573 generic.go:356] "Generic (PLEG): container finished" podID="135435c4-a57d-48a7-a631-db139fc0d5eb" containerID="3ca22a64a5e171adeb729b2b4f8b703952746589a5e1a1478924381af9e78225" exitCode=0 Sep 11 14:39:12.803051 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:12.803032 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" event={"ID":"135435c4-a57d-48a7-a631-db139fc0d5eb","Type":"ContainerDied","Data":"3ca22a64a5e171adeb729b2b4f8b703952746589a5e1a1478924381af9e78225"} Sep 11 14:39:12.803460 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:12.803445 2573 scope.go:117] "RemoveContainer" containerID="3ca22a64a5e171adeb729b2b4f8b703952746589a5e1a1478924381af9e78225" Sep 11 14:39:12.804661 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:12.804634 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:39:12.804778 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:12.804712 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:39:12.804778 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:12.804756 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:39:13.811150 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:13.811119 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" event={"ID":"135435c4-a57d-48a7-a631-db139fc0d5eb","Type":"ContainerStarted","Data":"81aa45da0223812c2bd96277f15c6857aa698458c6e64b0332e7431226e879a2"} Sep 11 14:39:14.552728 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:14.552685 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:39:14.553018 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:14.552900 2573 patch_prober.go:28] interesting pod/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 11 14:39:14.553018 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:14.552944 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" podUID="135435c4-a57d-48a7-a631-db139fc0d5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 11 14:39:15.552752 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:15.552714 2573 patch_prober.go:28] interesting pod/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 11 14:39:15.553150 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:15.552769 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" podUID="135435c4-a57d-48a7-a631-db139fc0d5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 11 14:39:16.552649 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:16.552606 2573 patch_prober.go:28] interesting pod/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 11 14:39:16.552881 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:16.552673 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" podUID="135435c4-a57d-48a7-a631-db139fc0d5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 11 14:39:17.553367 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:17.553332 2573 patch_prober.go:28] interesting pod/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 11 14:39:17.554217 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:17.553427 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" podUID="135435c4-a57d-48a7-a631-db139fc0d5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 11 14:39:18.552787 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:18.552751 2573 patch_prober.go:28] interesting pod/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Sep 11 14:39:18.553014 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:18.552805 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" podUID="135435c4-a57d-48a7-a631-db139fc0d5eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Sep 11 14:39:19.556332 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:19.556302 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:39:19.556798 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:19.556549 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:39:19.557150 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:19.557132 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:39:21.109358 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:21.109326 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4"] Sep 11 14:39:21.112049 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:21.112021 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4" Sep 11 14:39:21.114982 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:21.114962 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp8gp\"" Sep 11 14:39:21.115752 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:21.115735 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:39:21.126295 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:21.126270 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4"] Sep 11 14:39:21.213410 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:21.213375 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e39c37ba-300a-4818-909b-35fd55223ab8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4\" (UID: \"e39c37ba-300a-4818-909b-35fd55223ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4" Sep 11 14:39:21.314622 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:21.314576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e39c37ba-300a-4818-909b-35fd55223ab8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4\" (UID: \"e39c37ba-300a-4818-909b-35fd55223ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4" Sep 11 14:39:21.315349 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:21.315329 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e39c37ba-300a-4818-909b-35fd55223ab8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4\" (UID: \"e39c37ba-300a-4818-909b-35fd55223ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4" Sep 11 14:39:21.422776 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:21.422681 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4" Sep 11 14:39:21.709648 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:21.709574 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4"] Sep 11 14:39:21.715942 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:39:21.715915 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39c37ba_300a_4818_909b_35fd55223ab8.slice/crio-9253ec7592883d229630b50eca18d7d005fd6ba08280f558fb71c3d88dbb9494 WatchSource:0}: Error finding container 9253ec7592883d229630b50eca18d7d005fd6ba08280f558fb71c3d88dbb9494: Status 404 returned error can't find the container with id 9253ec7592883d229630b50eca18d7d005fd6ba08280f558fb71c3d88dbb9494 Sep 11 14:39:21.872887 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:21.872811 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4" event={"ID":"e39c37ba-300a-4818-909b-35fd55223ab8","Type":"ContainerStarted","Data":"9253ec7592883d229630b50eca18d7d005fd6ba08280f558fb71c3d88dbb9494"} Sep 11 14:39:22.880476 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:22.880442 2573 generic.go:356] "Generic (PLEG): container finished" podID="e39c37ba-300a-4818-909b-35fd55223ab8" containerID="b8b747ef807e4326afa16f47c51892b6c7fb16a6731f4a05fd79d1e832c66e61" exitCode=0 Sep 11 14:39:22.880865 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:22.880534 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4" event={"ID":"e39c37ba-300a-4818-909b-35fd55223ab8","Type":"ContainerDied","Data":"b8b747ef807e4326afa16f47c51892b6c7fb16a6731f4a05fd79d1e832c66e61"} Sep 11 14:39:24.020088 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:24.020066 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4" Sep 11 14:39:24.144555 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:24.144463 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e39c37ba-300a-4818-909b-35fd55223ab8-maas-api-service-ca\") pod \"e39c37ba-300a-4818-909b-35fd55223ab8\" (UID: \"e39c37ba-300a-4818-909b-35fd55223ab8\") " Sep 11 14:39:24.144882 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:24.144823 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39c37ba-300a-4818-909b-35fd55223ab8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e39c37ba-300a-4818-909b-35fd55223ab8" (UID: "e39c37ba-300a-4818-909b-35fd55223ab8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:39:24.245633 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:24.245595 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e39c37ba-300a-4818-909b-35fd55223ab8-maas-api-service-ca\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:39:24.893286 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:24.893258 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4" Sep 11 14:39:24.893451 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:24.893257 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4" event={"ID":"e39c37ba-300a-4818-909b-35fd55223ab8","Type":"ContainerDied","Data":"9253ec7592883d229630b50eca18d7d005fd6ba08280f558fb71c3d88dbb9494"} Sep 11 14:39:24.893451 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:24.893404 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9253ec7592883d229630b50eca18d7d005fd6ba08280f558fb71c3d88dbb9494" Sep 11 14:39:37.781985 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:37.781947 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4"] Sep 11 14:39:37.796611 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:37.796578 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2cabda-wnvm4"] Sep 11 14:39:38.187656 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:38.187623 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e39c37ba-300a-4818-909b-35fd55223ab8" path="/var/lib/kubelet/pods/e39c37ba-300a-4818-909b-35fd55223ab8/volumes" Sep 11 14:39:45.987596 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:45.987562 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b"] Sep 11 14:39:45.988016 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:45.987800 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" podUID="80ef2d79-ce56-416e-885c-bf5591f1082e" containerName="istio-proxy" containerID="cri-o://e2917142393a5f8a25d781b34fa68bb81488ce0577a71a7922398728c4e48069" gracePeriod=30 Sep 11 14:39:49.966401 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:49.966357 2573 patch_prober.go:28] interesting pod/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:39:49.966780 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:49.966429 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" podUID="80ef2d79-ce56-416e-885c-bf5591f1082e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:39:51.236938 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.236915 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:39:51.309667 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.309574 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zgcr\" (UniqueName: \"kubernetes.io/projected/80ef2d79-ce56-416e-885c-bf5591f1082e-kube-api-access-2zgcr\") pod \"80ef2d79-ce56-416e-885c-bf5591f1082e\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " Sep 11 14:39:51.309667 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.309620 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-token\") pod \"80ef2d79-ce56-416e-885c-bf5591f1082e\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " Sep 11 14:39:51.309667 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.309665 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-envoy\") pod \"80ef2d79-ce56-416e-885c-bf5591f1082e\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " Sep 11 14:39:51.309987 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.309708 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ef2d79-ce56-416e-885c-bf5591f1082e-istiod-ca-cert\") pod \"80ef2d79-ce56-416e-885c-bf5591f1082e\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " Sep 11 14:39:51.309987 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.309745 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-workload-certs\") pod \"80ef2d79-ce56-416e-885c-bf5591f1082e\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " Sep 11 14:39:51.309987 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.309816 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-podinfo\") pod \"80ef2d79-ce56-416e-885c-bf5591f1082e\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " Sep 11 14:39:51.309987 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.309872 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-data\") pod \"80ef2d79-ce56-416e-885c-bf5591f1082e\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " Sep 11 14:39:51.309987 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.309924 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-credential-socket\") pod \"80ef2d79-ce56-416e-885c-bf5591f1082e\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " Sep 11 14:39:51.309987 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.309953 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-workload-socket\") pod \"80ef2d79-ce56-416e-885c-bf5591f1082e\" (UID: \"80ef2d79-ce56-416e-885c-bf5591f1082e\") " Sep 11 14:39:51.310279 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.310087 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "80ef2d79-ce56-416e-885c-bf5591f1082e" (UID: "80ef2d79-ce56-416e-885c-bf5591f1082e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:39:51.310279 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.310152 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ef2d79-ce56-416e-885c-bf5591f1082e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "80ef2d79-ce56-416e-885c-bf5591f1082e" (UID: "80ef2d79-ce56-416e-885c-bf5591f1082e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:39:51.310391 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.310307 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "80ef2d79-ce56-416e-885c-bf5591f1082e" (UID: "80ef2d79-ce56-416e-885c-bf5591f1082e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:39:51.310463 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.310429 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-data" (OuterVolumeSpecName: "istio-data") pod "80ef2d79-ce56-416e-885c-bf5591f1082e" (UID: "80ef2d79-ce56-416e-885c-bf5591f1082e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:39:51.310463 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.310442 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "80ef2d79-ce56-416e-885c-bf5591f1082e" (UID: "80ef2d79-ce56-416e-885c-bf5591f1082e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:39:51.310589 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.310554 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ef2d79-ce56-416e-885c-bf5591f1082e-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:39:51.310589 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.310579 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:39:51.310670 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.310593 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:39:51.310670 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.310605 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:39:51.310670 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.310619 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:39:51.311965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.311938 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ef2d79-ce56-416e-885c-bf5591f1082e-kube-api-access-2zgcr" (OuterVolumeSpecName: "kube-api-access-2zgcr") pod "80ef2d79-ce56-416e-885c-bf5591f1082e" (UID: "80ef2d79-ce56-416e-885c-bf5591f1082e"). InnerVolumeSpecName "kube-api-access-2zgcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:39:51.312102 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.311990 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "80ef2d79-ce56-416e-885c-bf5591f1082e" (UID: "80ef2d79-ce56-416e-885c-bf5591f1082e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:39:51.312386 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.312362 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-token" (OuterVolumeSpecName: "istio-token") pod "80ef2d79-ce56-416e-885c-bf5591f1082e" (UID: "80ef2d79-ce56-416e-885c-bf5591f1082e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:39:51.312450 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.312369 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "80ef2d79-ce56-416e-885c-bf5591f1082e" (UID: "80ef2d79-ce56-416e-885c-bf5591f1082e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:39:51.412095 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.412056 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:39:51.412095 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.412088 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zgcr\" (UniqueName: \"kubernetes.io/projected/80ef2d79-ce56-416e-885c-bf5591f1082e-kube-api-access-2zgcr\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:39:51.412095 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.412098 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:39:51.412343 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:51.412109 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ef2d79-ce56-416e-885c-bf5591f1082e-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:39:52.060711 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:52.060673 2573 generic.go:356] "Generic (PLEG): container finished" podID="80ef2d79-ce56-416e-885c-bf5591f1082e" containerID="e2917142393a5f8a25d781b34fa68bb81488ce0577a71a7922398728c4e48069" exitCode=0 Sep 11 14:39:52.060928 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:52.060725 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" Sep 11 14:39:52.060928 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:52.060756 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" event={"ID":"80ef2d79-ce56-416e-885c-bf5591f1082e","Type":"ContainerDied","Data":"e2917142393a5f8a25d781b34fa68bb81488ce0577a71a7922398728c4e48069"} Sep 11 14:39:52.060928 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:52.060814 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b" event={"ID":"80ef2d79-ce56-416e-885c-bf5591f1082e","Type":"ContainerDied","Data":"de473a765c992df247a1e5784508abe450953f04c74444ee1691a90385414adf"} Sep 11 14:39:52.060928 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:52.060854 2573 scope.go:117] "RemoveContainer" containerID="e2917142393a5f8a25d781b34fa68bb81488ce0577a71a7922398728c4e48069" Sep 11 14:39:52.071029 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:52.071012 2573 scope.go:117] "RemoveContainer" containerID="9537c6fbdb3035931259ef2b6b1ce1ee3a9bd6ff0501fbd766e39fd93ad06424" Sep 11 14:39:52.079378 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:52.079357 2573 scope.go:117] "RemoveContainer" containerID="e2917142393a5f8a25d781b34fa68bb81488ce0577a71a7922398728c4e48069" Sep 11 14:39:52.079675 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:39:52.079656 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2917142393a5f8a25d781b34fa68bb81488ce0577a71a7922398728c4e48069\": container with ID starting with e2917142393a5f8a25d781b34fa68bb81488ce0577a71a7922398728c4e48069 not found: ID does not exist" containerID="e2917142393a5f8a25d781b34fa68bb81488ce0577a71a7922398728c4e48069" Sep 11 14:39:52.079739 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:52.079682 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2917142393a5f8a25d781b34fa68bb81488ce0577a71a7922398728c4e48069"} err="failed to get container status \"e2917142393a5f8a25d781b34fa68bb81488ce0577a71a7922398728c4e48069\": rpc error: code = NotFound desc = could not find container \"e2917142393a5f8a25d781b34fa68bb81488ce0577a71a7922398728c4e48069\": container with ID starting with e2917142393a5f8a25d781b34fa68bb81488ce0577a71a7922398728c4e48069 not found: ID does not exist" Sep 11 14:39:52.079739 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:52.079700 2573 scope.go:117] "RemoveContainer" containerID="9537c6fbdb3035931259ef2b6b1ce1ee3a9bd6ff0501fbd766e39fd93ad06424" Sep 11 14:39:52.079965 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:39:52.079949 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9537c6fbdb3035931259ef2b6b1ce1ee3a9bd6ff0501fbd766e39fd93ad06424\": container with ID starting with 9537c6fbdb3035931259ef2b6b1ce1ee3a9bd6ff0501fbd766e39fd93ad06424 not found: ID does not exist" containerID="9537c6fbdb3035931259ef2b6b1ce1ee3a9bd6ff0501fbd766e39fd93ad06424" Sep 11 14:39:52.080012 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:52.079969 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9537c6fbdb3035931259ef2b6b1ce1ee3a9bd6ff0501fbd766e39fd93ad06424"} err="failed to get container status \"9537c6fbdb3035931259ef2b6b1ce1ee3a9bd6ff0501fbd766e39fd93ad06424\": rpc error: code = NotFound desc = could not find container \"9537c6fbdb3035931259ef2b6b1ce1ee3a9bd6ff0501fbd766e39fd93ad06424\": container with ID starting with 9537c6fbdb3035931259ef2b6b1ce1ee3a9bd6ff0501fbd766e39fd93ad06424 not found: ID does not exist" Sep 11 14:39:52.091450 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:52.091407 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b"] Sep 11 14:39:52.094150 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:52.094129 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2cabda-openshift-default-6554cf7d8b-q4m5b"] Sep 11 14:39:52.187247 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:39:52.187212 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80ef2d79-ce56-416e-885c-bf5591f1082e" path="/var/lib/kubelet/pods/80ef2d79-ce56-416e-885c-bf5591f1082e/volumes" Sep 11 14:40:06.854972 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.854936 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp"] Sep 11 14:40:06.855581 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.855563 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ef2d79-ce56-416e-885c-bf5591f1082e" containerName="istio-proxy" Sep 11 14:40:06.855647 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.855583 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ef2d79-ce56-416e-885c-bf5591f1082e" containerName="istio-proxy" Sep 11 14:40:06.855647 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.855596 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ef2d79-ce56-416e-885c-bf5591f1082e" containerName="istio-proxy" Sep 11 14:40:06.855647 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.855605 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ef2d79-ce56-416e-885c-bf5591f1082e" containerName="istio-proxy" Sep 11 14:40:06.855647 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.855620 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e39c37ba-300a-4818-909b-35fd55223ab8" containerName="revoke-keys" Sep 11 14:40:06.855647 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.855629 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39c37ba-300a-4818-909b-35fd55223ab8" containerName="revoke-keys" Sep 11 14:40:06.855826 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.855747 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ef2d79-ce56-416e-885c-bf5591f1082e" containerName="istio-proxy" Sep 11 14:40:06.855826 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.855766 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e39c37ba-300a-4818-909b-35fd55223ab8" containerName="revoke-keys" Sep 11 14:40:06.855826 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.855776 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ef2d79-ce56-416e-885c-bf5591f1082e" containerName="istio-proxy" Sep 11 14:40:06.859557 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.859540 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.863542 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.863521 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a6c3fc-openshift-default-dockercfg-fzpxv\"" Sep 11 14:40:06.872704 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.872679 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-token\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.872793 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.872726 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-envoy\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.872793 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.872753 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-workload-certs\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.872894 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.872860 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-podinfo\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.872931 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.872911 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b3b60b-cb0a-48bd-982b-ff12f647752a-istiod-ca-cert\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.872965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.872938 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7c76\" (UniqueName: \"kubernetes.io/projected/82b3b60b-cb0a-48bd-982b-ff12f647752a-kube-api-access-l7c76\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.873006 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.872972 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-workload-socket\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.873040 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.873021 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-credential-socket\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.873074 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.873047 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-data\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.901554 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.901445 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp"] Sep 11 14:40:06.974436 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.974408 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-podinfo\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.974640 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.974461 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b3b60b-cb0a-48bd-982b-ff12f647752a-istiod-ca-cert\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.974640 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.974482 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7c76\" (UniqueName: \"kubernetes.io/projected/82b3b60b-cb0a-48bd-982b-ff12f647752a-kube-api-access-l7c76\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.974640 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.974504 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-workload-socket\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.974640 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.974523 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-credential-socket\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.974640 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.974540 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-data\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.974640 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.974576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-token\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.974640 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.974613 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-envoy\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.974640 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.974637 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-workload-certs\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.975111 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.975083 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-workload-socket\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.975164 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.975141 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-credential-socket\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.975209 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.975185 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-data\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.975278 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.975256 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b3b60b-cb0a-48bd-982b-ff12f647752a-istiod-ca-cert\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.975434 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.975414 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-workload-certs\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.977052 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.977025 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-envoy\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.977271 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.977252 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-podinfo\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.996763 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.996737 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-token\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:06.997262 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:06.997243 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7c76\" (UniqueName: \"kubernetes.io/projected/82b3b60b-cb0a-48bd-982b-ff12f647752a-kube-api-access-l7c76\") pod \"e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:07.173287 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:07.173206 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:07.479377 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:07.479347 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp"] Sep 11 14:40:07.480788 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:40:07.480761 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b3b60b_cb0a_48bd_982b_ff12f647752a.slice/crio-de2dd5d49dd8efe64b3eb46a50cf4090ba6caf223384a733c6c27dd0caabebd8 WatchSource:0}: Error finding container de2dd5d49dd8efe64b3eb46a50cf4090ba6caf223384a733c6c27dd0caabebd8: Status 404 returned error can't find the container with id de2dd5d49dd8efe64b3eb46a50cf4090ba6caf223384a733c6c27dd0caabebd8 Sep 11 14:40:07.483014 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:07.482984 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:40:07.483068 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:07.483046 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:40:07.483105 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:07.483074 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:40:08.165714 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:08.165677 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" event={"ID":"82b3b60b-cb0a-48bd-982b-ff12f647752a","Type":"ContainerStarted","Data":"ef4ac7e17c497702bd2f36845f7a2782b2c6b122a307ae0c2af7f4efbd348544"} Sep 11 14:40:08.166129 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:08.165739 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" event={"ID":"82b3b60b-cb0a-48bd-982b-ff12f647752a","Type":"ContainerStarted","Data":"de2dd5d49dd8efe64b3eb46a50cf4090ba6caf223384a733c6c27dd0caabebd8"} Sep 11 14:40:08.173739 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:08.173706 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:08.213661 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:08.213630 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:08.223821 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:08.223768 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" podStartSLOduration=2.223751796 podStartE2EDuration="2.223751796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:08.220540956 +0000 UTC m=+1520.557509148" watchObservedRunningTime="2026-09-11 14:40:08.223751796 +0000 UTC m=+1520.560719986" Sep 11 14:40:09.172405 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:09.172369 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:09.173219 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:09.173197 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:11.957030 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:11.956955 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926"] Sep 11 14:40:11.957395 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:11.957198 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" podUID="135435c4-a57d-48a7-a631-db139fc0d5eb" containerName="istio-proxy" containerID="cri-o://81aa45da0223812c2bd96277f15c6857aa698458c6e64b0332e7431226e879a2" gracePeriod=30 Sep 11 14:40:17.208068 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.208044 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:40:17.223038 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.222972 2573 generic.go:356] "Generic (PLEG): container finished" podID="135435c4-a57d-48a7-a631-db139fc0d5eb" containerID="81aa45da0223812c2bd96277f15c6857aa698458c6e64b0332e7431226e879a2" exitCode=0 Sep 11 14:40:17.223038 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.223009 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" Sep 11 14:40:17.223188 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.223059 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" event={"ID":"135435c4-a57d-48a7-a631-db139fc0d5eb","Type":"ContainerDied","Data":"81aa45da0223812c2bd96277f15c6857aa698458c6e64b0332e7431226e879a2"} Sep 11 14:40:17.223188 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.223108 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926" event={"ID":"135435c4-a57d-48a7-a631-db139fc0d5eb","Type":"ContainerDied","Data":"e1af748d766b8ff012a9c839452d5feb03e5ae05ba28b4f8e081889a054fbe9a"} Sep 11 14:40:17.223188 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.223132 2573 scope.go:117] "RemoveContainer" containerID="81aa45da0223812c2bd96277f15c6857aa698458c6e64b0332e7431226e879a2" Sep 11 14:40:17.233236 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.233216 2573 scope.go:117] "RemoveContainer" containerID="3ca22a64a5e171adeb729b2b4f8b703952746589a5e1a1478924381af9e78225" Sep 11 14:40:17.244549 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.243719 2573 scope.go:117] "RemoveContainer" containerID="81aa45da0223812c2bd96277f15c6857aa698458c6e64b0332e7431226e879a2" Sep 11 14:40:17.244549 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:40:17.244055 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81aa45da0223812c2bd96277f15c6857aa698458c6e64b0332e7431226e879a2\": container with ID starting with 81aa45da0223812c2bd96277f15c6857aa698458c6e64b0332e7431226e879a2 not found: ID does not exist" containerID="81aa45da0223812c2bd96277f15c6857aa698458c6e64b0332e7431226e879a2" Sep 11 14:40:17.244549 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.244088 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81aa45da0223812c2bd96277f15c6857aa698458c6e64b0332e7431226e879a2"} err="failed to get container status \"81aa45da0223812c2bd96277f15c6857aa698458c6e64b0332e7431226e879a2\": rpc error: code = NotFound desc = could not find container \"81aa45da0223812c2bd96277f15c6857aa698458c6e64b0332e7431226e879a2\": container with ID starting with 81aa45da0223812c2bd96277f15c6857aa698458c6e64b0332e7431226e879a2 not found: ID does not exist" Sep 11 14:40:17.244549 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.244111 2573 scope.go:117] "RemoveContainer" containerID="3ca22a64a5e171adeb729b2b4f8b703952746589a5e1a1478924381af9e78225" Sep 11 14:40:17.244549 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:40:17.244397 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca22a64a5e171adeb729b2b4f8b703952746589a5e1a1478924381af9e78225\": container with ID starting with 3ca22a64a5e171adeb729b2b4f8b703952746589a5e1a1478924381af9e78225 not found: ID does not exist" containerID="3ca22a64a5e171adeb729b2b4f8b703952746589a5e1a1478924381af9e78225" Sep 11 14:40:17.244549 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.244422 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca22a64a5e171adeb729b2b4f8b703952746589a5e1a1478924381af9e78225"} err="failed to get container status \"3ca22a64a5e171adeb729b2b4f8b703952746589a5e1a1478924381af9e78225\": rpc error: code = NotFound desc = could not find container \"3ca22a64a5e171adeb729b2b4f8b703952746589a5e1a1478924381af9e78225\": container with ID starting with 3ca22a64a5e171adeb729b2b4f8b703952746589a5e1a1478924381af9e78225 not found: ID does not exist" Sep 11 14:40:17.287511 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.287484 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-data\") pod \"135435c4-a57d-48a7-a631-db139fc0d5eb\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " Sep 11 14:40:17.287652 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.287572 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-workload-socket\") pod \"135435c4-a57d-48a7-a631-db139fc0d5eb\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " Sep 11 14:40:17.287652 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.287606 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-token\") pod \"135435c4-a57d-48a7-a631-db139fc0d5eb\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " Sep 11 14:40:17.287652 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.287648 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-credential-socket\") pod \"135435c4-a57d-48a7-a631-db139fc0d5eb\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " Sep 11 14:40:17.287794 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.287692 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-podinfo\") pod \"135435c4-a57d-48a7-a631-db139fc0d5eb\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " Sep 11 14:40:17.287794 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.287714 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135435c4-a57d-48a7-a631-db139fc0d5eb-istiod-ca-cert\") pod \"135435c4-a57d-48a7-a631-db139fc0d5eb\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " Sep 11 14:40:17.287794 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.287742 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-workload-certs\") pod \"135435c4-a57d-48a7-a631-db139fc0d5eb\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " Sep 11 14:40:17.287980 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.287783 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "135435c4-a57d-48a7-a631-db139fc0d5eb" (UID: "135435c4-a57d-48a7-a631-db139fc0d5eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:40:17.287980 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.287792 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "135435c4-a57d-48a7-a631-db139fc0d5eb" (UID: "135435c4-a57d-48a7-a631-db139fc0d5eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:40:17.287980 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.287827 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxkx6\" (UniqueName: \"kubernetes.io/projected/135435c4-a57d-48a7-a631-db139fc0d5eb-kube-api-access-fxkx6\") pod \"135435c4-a57d-48a7-a631-db139fc0d5eb\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " Sep 11 14:40:17.287980 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.287904 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-envoy\") pod \"135435c4-a57d-48a7-a631-db139fc0d5eb\" (UID: \"135435c4-a57d-48a7-a631-db139fc0d5eb\") " Sep 11 14:40:17.288200 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.288005 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "135435c4-a57d-48a7-a631-db139fc0d5eb" (UID: "135435c4-a57d-48a7-a631-db139fc0d5eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:40:17.288200 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.288023 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "135435c4-a57d-48a7-a631-db139fc0d5eb" (UID: "135435c4-a57d-48a7-a631-db139fc0d5eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:40:17.288200 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.288145 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135435c4-a57d-48a7-a631-db139fc0d5eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "135435c4-a57d-48a7-a631-db139fc0d5eb" (UID: "135435c4-a57d-48a7-a631-db139fc0d5eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:40:17.288728 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.288632 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:17.288728 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.288658 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:17.288728 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.288672 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135435c4-a57d-48a7-a631-db139fc0d5eb-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:17.288728 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.288687 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:17.288728 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.288699 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:17.290501 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.290429 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135435c4-a57d-48a7-a631-db139fc0d5eb-kube-api-access-fxkx6" (OuterVolumeSpecName: "kube-api-access-fxkx6") pod "135435c4-a57d-48a7-a631-db139fc0d5eb" (UID: "135435c4-a57d-48a7-a631-db139fc0d5eb"). InnerVolumeSpecName "kube-api-access-fxkx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:40:17.290629 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.290536 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "135435c4-a57d-48a7-a631-db139fc0d5eb" (UID: "135435c4-a57d-48a7-a631-db139fc0d5eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:40:17.290717 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.290639 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "135435c4-a57d-48a7-a631-db139fc0d5eb" (UID: "135435c4-a57d-48a7-a631-db139fc0d5eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:40:17.290782 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.290650 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "135435c4-a57d-48a7-a631-db139fc0d5eb" (UID: "135435c4-a57d-48a7-a631-db139fc0d5eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:40:17.389493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.389453 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:17.389493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.389491 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:17.389493 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.389501 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxkx6\" (UniqueName: \"kubernetes.io/projected/135435c4-a57d-48a7-a631-db139fc0d5eb-kube-api-access-fxkx6\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:17.389705 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.389510 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135435c4-a57d-48a7-a631-db139fc0d5eb-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:17.556128 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.556099 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926"] Sep 11 14:40:17.580019 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:17.579993 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-96e743-openshift-default-5cc55d597d-zb926"] Sep 11 14:40:18.187731 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:18.187702 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="135435c4-a57d-48a7-a631-db139fc0d5eb" path="/var/lib/kubelet/pods/135435c4-a57d-48a7-a631-db139fc0d5eb/volumes" Sep 11 14:40:31.504968 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.504932 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6"] Sep 11 14:40:31.505449 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.505349 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="135435c4-a57d-48a7-a631-db139fc0d5eb" containerName="istio-proxy" Sep 11 14:40:31.505449 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.505360 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="135435c4-a57d-48a7-a631-db139fc0d5eb" containerName="istio-proxy" Sep 11 14:40:31.505449 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.505372 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="135435c4-a57d-48a7-a631-db139fc0d5eb" containerName="istio-proxy" Sep 11 14:40:31.505449 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.505377 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="135435c4-a57d-48a7-a631-db139fc0d5eb" containerName="istio-proxy" Sep 11 14:40:31.505660 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.505459 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="135435c4-a57d-48a7-a631-db139fc0d5eb" containerName="istio-proxy" Sep 11 14:40:31.505660 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.505469 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="135435c4-a57d-48a7-a631-db139fc0d5eb" containerName="istio-proxy" Sep 11 14:40:31.508715 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.508697 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.511556 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.511534 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-625d10-openshift-default-dockercfg-ct785\"" Sep 11 14:40:31.528416 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.528388 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6"] Sep 11 14:40:31.640674 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.640637 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-workload-certs\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.640885 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.640726 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-podinfo\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.640885 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.640782 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-envoy\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.640885 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.640806 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d9539d-aec8-44aa-a184-16fc04d21c90-istiod-ca-cert\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.640885 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.640865 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-credential-socket\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.647738 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.640958 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-token\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.647738 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.641430 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4dm5\" (UniqueName: \"kubernetes.io/projected/42d9539d-aec8-44aa-a184-16fc04d21c90-kube-api-access-n4dm5\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.647738 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.641571 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-data\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.647738 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.641638 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-workload-socket\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.742650 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.742612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-credential-socket\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.742862 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.742674 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-token\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.742862 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.742696 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4dm5\" (UniqueName: \"kubernetes.io/projected/42d9539d-aec8-44aa-a184-16fc04d21c90-kube-api-access-n4dm5\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.742862 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.742734 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-data\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.742862 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.742762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-workload-socket\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.742862 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.742799 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-workload-certs\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.743155 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.742904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-podinfo\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.743155 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.742950 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-envoy\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.743155 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.743004 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d9539d-aec8-44aa-a184-16fc04d21c90-istiod-ca-cert\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.743155 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.743132 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-credential-socket\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.743379 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.743358 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-workload-socket\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.743481 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.743458 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-workload-certs\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.743598 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.743575 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-data\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.743721 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.743704 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d9539d-aec8-44aa-a184-16fc04d21c90-istiod-ca-cert\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.745627 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.745606 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-envoy\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.746006 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.745988 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-podinfo\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.753517 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.753486 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4dm5\" (UniqueName: \"kubernetes.io/projected/42d9539d-aec8-44aa-a184-16fc04d21c90-kube-api-access-n4dm5\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.753977 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.753953 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-token\") pod \"e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:31.820545 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:31.820467 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:32.108050 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:32.108022 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6"] Sep 11 14:40:32.109919 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:40:32.109887 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d9539d_aec8_44aa_a184_16fc04d21c90.slice/crio-8f9e7a2b14e6059c8fb380b93ed625fa9f4ca9d8755362ee63da53ec0678a238 WatchSource:0}: Error finding container 8f9e7a2b14e6059c8fb380b93ed625fa9f4ca9d8755362ee63da53ec0678a238: Status 404 returned error can't find the container with id 8f9e7a2b14e6059c8fb380b93ed625fa9f4ca9d8755362ee63da53ec0678a238 Sep 11 14:40:32.112502 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:32.112457 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:40:32.112596 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:32.112551 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:40:32.112641 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:32.112598 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:40:32.324454 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:32.324154 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" event={"ID":"42d9539d-aec8-44aa-a184-16fc04d21c90","Type":"ContainerStarted","Data":"fe89c6a340ec5de2714a8f8dd4fa5f6a028809bb1087f3897f0543b5d097c6b1"} Sep 11 14:40:32.324454 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:32.324223 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" event={"ID":"42d9539d-aec8-44aa-a184-16fc04d21c90","Type":"ContainerStarted","Data":"8f9e7a2b14e6059c8fb380b93ed625fa9f4ca9d8755362ee63da53ec0678a238"} Sep 11 14:40:32.327286 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:32.327255 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk"] Sep 11 14:40:32.327440 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:32.327421 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" podUID="1e9027ad-28a5-44d9-b935-fcf83bee47de" containerName="istio-proxy" containerID="cri-o://01768a1d5ffff7fc6c3c1cb5decb2b8234918a0c2dd6dfa18b516928044bf058" gracePeriod=30 Sep 11 14:40:32.365104 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:32.365016 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" podStartSLOduration=1.365000383 podStartE2EDuration="1.365000383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:32.362577935 +0000 UTC m=+1544.699546149" watchObservedRunningTime="2026-09-11 14:40:32.365000383 +0000 UTC m=+1544.701968573" Sep 11 14:40:32.821566 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:32.821530 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:33.332116 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:33.332085 2573 generic.go:356] "Generic (PLEG): container finished" podID="82b3b60b-cb0a-48bd-982b-ff12f647752a" containerID="ef4ac7e17c497702bd2f36845f7a2782b2c6b122a307ae0c2af7f4efbd348544" exitCode=0 Sep 11 14:40:33.332295 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:33.332165 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" event={"ID":"82b3b60b-cb0a-48bd-982b-ff12f647752a","Type":"ContainerDied","Data":"ef4ac7e17c497702bd2f36845f7a2782b2c6b122a307ae0c2af7f4efbd348544"} Sep 11 14:40:33.332608 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:33.332594 2573 scope.go:117] "RemoveContainer" containerID="ef4ac7e17c497702bd2f36845f7a2782b2c6b122a307ae0c2af7f4efbd348544" Sep 11 14:40:33.333678 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:33.333649 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:40:33.333787 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:33.333729 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:40:33.333787 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:33.333772 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:40:33.831610 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:33.831345 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:34.346106 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:34.346037 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" event={"ID":"82b3b60b-cb0a-48bd-982b-ff12f647752a","Type":"ContainerStarted","Data":"791e7787c9029add5f358537f6ac78ced1a3ac7bb967bd059b88230907fc701c"} Sep 11 14:40:34.346312 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:34.346125 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:34.351074 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:34.351044 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:35.174440 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:35.174394 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:35.175644 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:35.175618 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Sep 11 14:40:35.175716 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:35.175671 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" podUID="82b3b60b-cb0a-48bd-982b-ff12f647752a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Sep 11 14:40:36.174173 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:36.174136 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Sep 11 14:40:36.174369 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:36.174209 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" podUID="82b3b60b-cb0a-48bd-982b-ff12f647752a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Sep 11 14:40:37.173612 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.173575 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Sep 11 14:40:37.174035 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.173629 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" podUID="82b3b60b-cb0a-48bd-982b-ff12f647752a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Sep 11 14:40:37.619529 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.619505 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:40:37.707114 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707035 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-workload-certs\") pod \"1e9027ad-28a5-44d9-b935-fcf83bee47de\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " Sep 11 14:40:37.707114 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707140 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-workload-socket\") pod \"1e9027ad-28a5-44d9-b935-fcf83bee47de\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " Sep 11 14:40:37.707372 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707184 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-data\") pod \"1e9027ad-28a5-44d9-b935-fcf83bee47de\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " Sep 11 14:40:37.707372 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707211 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bghb2\" (UniqueName: \"kubernetes.io/projected/1e9027ad-28a5-44d9-b935-fcf83bee47de-kube-api-access-bghb2\") pod \"1e9027ad-28a5-44d9-b935-fcf83bee47de\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " Sep 11 14:40:37.707372 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707250 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e9027ad-28a5-44d9-b935-fcf83bee47de-istiod-ca-cert\") pod \"1e9027ad-28a5-44d9-b935-fcf83bee47de\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " Sep 11 14:40:37.707372 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707307 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e9027ad-28a5-44d9-b935-fcf83bee47de" (UID: "1e9027ad-28a5-44d9-b935-fcf83bee47de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:40:37.707602 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707377 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-token\") pod \"1e9027ad-28a5-44d9-b935-fcf83bee47de\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " Sep 11 14:40:37.707602 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707423 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-credential-socket\") pod \"1e9027ad-28a5-44d9-b935-fcf83bee47de\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " Sep 11 14:40:37.707602 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707372 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e9027ad-28a5-44d9-b935-fcf83bee47de" (UID: "1e9027ad-28a5-44d9-b935-fcf83bee47de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:40:37.707602 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707471 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-podinfo\") pod \"1e9027ad-28a5-44d9-b935-fcf83bee47de\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " Sep 11 14:40:37.707602 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707515 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-envoy\") pod \"1e9027ad-28a5-44d9-b935-fcf83bee47de\" (UID: \"1e9027ad-28a5-44d9-b935-fcf83bee47de\") " Sep 11 14:40:37.707877 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707513 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e9027ad-28a5-44d9-b935-fcf83bee47de" (UID: "1e9027ad-28a5-44d9-b935-fcf83bee47de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:40:37.707877 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707652 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e9027ad-28a5-44d9-b935-fcf83bee47de" (UID: "1e9027ad-28a5-44d9-b935-fcf83bee47de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:40:37.707877 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707686 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9027ad-28a5-44d9-b935-fcf83bee47de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e9027ad-28a5-44d9-b935-fcf83bee47de" (UID: "1e9027ad-28a5-44d9-b935-fcf83bee47de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:40:37.708026 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707940 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:37.708026 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707961 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e9027ad-28a5-44d9-b935-fcf83bee47de-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:37.708026 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707974 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:37.708026 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.707988 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:37.708026 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.708000 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:37.709544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.709517 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9027ad-28a5-44d9-b935-fcf83bee47de-kube-api-access-bghb2" (OuterVolumeSpecName: "kube-api-access-bghb2") pod "1e9027ad-28a5-44d9-b935-fcf83bee47de" (UID: "1e9027ad-28a5-44d9-b935-fcf83bee47de"). InnerVolumeSpecName "kube-api-access-bghb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:40:37.709544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.709513 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e9027ad-28a5-44d9-b935-fcf83bee47de" (UID: "1e9027ad-28a5-44d9-b935-fcf83bee47de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:40:37.709797 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.709783 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e9027ad-28a5-44d9-b935-fcf83bee47de" (UID: "1e9027ad-28a5-44d9-b935-fcf83bee47de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:40:37.709887 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.709791 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e9027ad-28a5-44d9-b935-fcf83bee47de" (UID: "1e9027ad-28a5-44d9-b935-fcf83bee47de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:40:37.809483 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.809444 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bghb2\" (UniqueName: \"kubernetes.io/projected/1e9027ad-28a5-44d9-b935-fcf83bee47de-kube-api-access-bghb2\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:37.809483 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.809479 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:37.809483 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.809489 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:37.809717 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:37.809497 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e9027ad-28a5-44d9-b935-fcf83bee47de-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:38.174404 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:38.174359 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Sep 11 14:40:38.174772 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:38.174427 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" podUID="82b3b60b-cb0a-48bd-982b-ff12f647752a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Sep 11 14:40:38.379810 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:38.379767 2573 generic.go:356] "Generic (PLEG): container finished" podID="1e9027ad-28a5-44d9-b935-fcf83bee47de" containerID="01768a1d5ffff7fc6c3c1cb5decb2b8234918a0c2dd6dfa18b516928044bf058" exitCode=0 Sep 11 14:40:38.380010 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:38.379884 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" event={"ID":"1e9027ad-28a5-44d9-b935-fcf83bee47de","Type":"ContainerDied","Data":"01768a1d5ffff7fc6c3c1cb5decb2b8234918a0c2dd6dfa18b516928044bf058"} Sep 11 14:40:38.380010 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:38.379936 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" event={"ID":"1e9027ad-28a5-44d9-b935-fcf83bee47de","Type":"ContainerDied","Data":"c33009ffa480f2f3bb15eadd51417e190537ab06ec741180f449b5983b501fdc"} Sep 11 14:40:38.380010 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:38.379966 2573 scope.go:117] "RemoveContainer" containerID="01768a1d5ffff7fc6c3c1cb5decb2b8234918a0c2dd6dfa18b516928044bf058" Sep 11 14:40:38.380237 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:38.380142 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk" Sep 11 14:40:38.392668 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:38.392637 2573 scope.go:117] "RemoveContainer" containerID="83a99cbd84aea532d0d03b55fd632dd0b3ad6f2b822582d3e3bfd7f01eff75f3" Sep 11 14:40:38.401681 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:38.401660 2573 scope.go:117] "RemoveContainer" containerID="01768a1d5ffff7fc6c3c1cb5decb2b8234918a0c2dd6dfa18b516928044bf058" Sep 11 14:40:38.401967 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:40:38.401941 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01768a1d5ffff7fc6c3c1cb5decb2b8234918a0c2dd6dfa18b516928044bf058\": container with ID starting with 01768a1d5ffff7fc6c3c1cb5decb2b8234918a0c2dd6dfa18b516928044bf058 not found: ID does not exist" containerID="01768a1d5ffff7fc6c3c1cb5decb2b8234918a0c2dd6dfa18b516928044bf058" Sep 11 14:40:38.402030 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:38.401976 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01768a1d5ffff7fc6c3c1cb5decb2b8234918a0c2dd6dfa18b516928044bf058"} err="failed to get container status \"01768a1d5ffff7fc6c3c1cb5decb2b8234918a0c2dd6dfa18b516928044bf058\": rpc error: code = NotFound desc = could not find container \"01768a1d5ffff7fc6c3c1cb5decb2b8234918a0c2dd6dfa18b516928044bf058\": container with ID starting with 01768a1d5ffff7fc6c3c1cb5decb2b8234918a0c2dd6dfa18b516928044bf058 not found: ID does not exist" Sep 11 14:40:38.402030 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:38.401992 2573 scope.go:117] "RemoveContainer" containerID="83a99cbd84aea532d0d03b55fd632dd0b3ad6f2b822582d3e3bfd7f01eff75f3" Sep 11 14:40:38.402237 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:40:38.402222 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a99cbd84aea532d0d03b55fd632dd0b3ad6f2b822582d3e3bfd7f01eff75f3\": container with ID starting with 83a99cbd84aea532d0d03b55fd632dd0b3ad6f2b822582d3e3bfd7f01eff75f3 not found: ID does not exist" containerID="83a99cbd84aea532d0d03b55fd632dd0b3ad6f2b822582d3e3bfd7f01eff75f3" Sep 11 14:40:38.402293 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:38.402242 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a99cbd84aea532d0d03b55fd632dd0b3ad6f2b822582d3e3bfd7f01eff75f3"} err="failed to get container status \"83a99cbd84aea532d0d03b55fd632dd0b3ad6f2b822582d3e3bfd7f01eff75f3\": rpc error: code = NotFound desc = could not find container \"83a99cbd84aea532d0d03b55fd632dd0b3ad6f2b822582d3e3bfd7f01eff75f3\": container with ID starting with 83a99cbd84aea532d0d03b55fd632dd0b3ad6f2b822582d3e3bfd7f01eff75f3 not found: ID does not exist" Sep 11 14:40:38.408774 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:38.408750 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk"] Sep 11 14:40:38.412661 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:38.412639 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b934e8-openshift-default-6cbcc565bd-q25tk"] Sep 11 14:40:39.173172 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:39.173131 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:39.173650 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:39.173623 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Sep 11 14:40:39.173713 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:39.173682 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" podUID="82b3b60b-cb0a-48bd-982b-ff12f647752a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Sep 11 14:40:40.178338 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:40.178307 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:40.179482 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:40.179462 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:40:40.187529 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:40.187504 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e9027ad-28a5-44d9-b935-fcf83bee47de" path="/var/lib/kubelet/pods/1e9027ad-28a5-44d9-b935-fcf83bee47de/volumes" Sep 11 14:40:53.135187 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:53.135153 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k"] Sep 11 14:40:53.135662 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:53.135368 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" podUID="d9ba867f-8e72-49f6-8f27-7cf8d0411f48" containerName="istio-proxy" containerID="cri-o://988e62bb18bc28871ad9ad7f8c84dc8f25f413ee81191fe98a9fe172726b8d36" gracePeriod=30 Sep 11 14:40:53.487637 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:53.487553 2573 generic.go:356] "Generic (PLEG): container finished" podID="42d9539d-aec8-44aa-a184-16fc04d21c90" containerID="fe89c6a340ec5de2714a8f8dd4fa5f6a028809bb1087f3897f0543b5d097c6b1" exitCode=0 Sep 11 14:40:53.487637 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:53.487626 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" event={"ID":"42d9539d-aec8-44aa-a184-16fc04d21c90","Type":"ContainerDied","Data":"fe89c6a340ec5de2714a8f8dd4fa5f6a028809bb1087f3897f0543b5d097c6b1"} Sep 11 14:40:53.488059 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:53.488045 2573 scope.go:117] "RemoveContainer" containerID="fe89c6a340ec5de2714a8f8dd4fa5f6a028809bb1087f3897f0543b5d097c6b1" Sep 11 14:40:53.489178 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:53.489149 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:40:53.489270 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:53.489213 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:40:53.489270 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:53.489255 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:40:54.496149 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:54.496114 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" event={"ID":"42d9539d-aec8-44aa-a184-16fc04d21c90","Type":"ContainerStarted","Data":"97b85e4fc5cb953a591a6b7bcada760d24a99cc5417ecd709f1ff97b43dbd133"} Sep 11 14:40:54.821017 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:54.820934 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:54.821227 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:54.821206 2573 patch_prober.go:28] interesting pod/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 11 14:40:54.821272 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:54.821256 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" podUID="42d9539d-aec8-44aa-a184-16fc04d21c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 11 14:40:55.821019 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:55.820986 2573 patch_prober.go:28] interesting pod/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 11 14:40:55.821413 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:55.821054 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" podUID="42d9539d-aec8-44aa-a184-16fc04d21c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 11 14:40:56.821575 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:56.821534 2573 patch_prober.go:28] interesting pod/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 11 14:40:56.822142 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:56.821614 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" podUID="42d9539d-aec8-44aa-a184-16fc04d21c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 11 14:40:57.821116 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:57.821078 2573 patch_prober.go:28] interesting pod/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 11 14:40:57.821371 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:57.821145 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" podUID="42d9539d-aec8-44aa-a184-16fc04d21c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 11 14:40:58.414207 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.414153 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:40:58.456422 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.456386 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-workload-socket\") pod \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " Sep 11 14:40:58.456604 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.456511 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-podinfo\") pod \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " Sep 11 14:40:58.456604 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.456539 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdvd5\" (UniqueName: \"kubernetes.io/projected/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-kube-api-access-xdvd5\") pod \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " Sep 11 14:40:58.456604 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.456568 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-workload-certs\") pod \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " Sep 11 14:40:58.456604 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.456600 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istiod-ca-cert\") pod \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " Sep 11 14:40:58.456826 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.456623 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-data\") pod \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " Sep 11 14:40:58.456826 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.456648 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-token\") pod \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " Sep 11 14:40:58.456826 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.456685 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9ba867f-8e72-49f6-8f27-7cf8d0411f48" (UID: "d9ba867f-8e72-49f6-8f27-7cf8d0411f48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:40:58.456826 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.456698 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-credential-socket\") pod \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " Sep 11 14:40:58.457060 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.456861 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9ba867f-8e72-49f6-8f27-7cf8d0411f48" (UID: "d9ba867f-8e72-49f6-8f27-7cf8d0411f48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:40:58.457158 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.457135 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:58.457223 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.457163 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:58.457223 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.457163 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9ba867f-8e72-49f6-8f27-7cf8d0411f48" (UID: "d9ba867f-8e72-49f6-8f27-7cf8d0411f48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:40:58.457381 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.457356 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9ba867f-8e72-49f6-8f27-7cf8d0411f48" (UID: "d9ba867f-8e72-49f6-8f27-7cf8d0411f48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:40:58.457509 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.457488 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9ba867f-8e72-49f6-8f27-7cf8d0411f48" (UID: "d9ba867f-8e72-49f6-8f27-7cf8d0411f48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:40:58.459014 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.458987 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9ba867f-8e72-49f6-8f27-7cf8d0411f48" (UID: "d9ba867f-8e72-49f6-8f27-7cf8d0411f48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:40:58.459332 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.459301 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-kube-api-access-xdvd5" (OuterVolumeSpecName: "kube-api-access-xdvd5") pod "d9ba867f-8e72-49f6-8f27-7cf8d0411f48" (UID: "d9ba867f-8e72-49f6-8f27-7cf8d0411f48"). InnerVolumeSpecName "kube-api-access-xdvd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:40:58.459508 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.459484 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9ba867f-8e72-49f6-8f27-7cf8d0411f48" (UID: "d9ba867f-8e72-49f6-8f27-7cf8d0411f48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:40:58.551998 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.551962 2573 generic.go:356] "Generic (PLEG): container finished" podID="d9ba867f-8e72-49f6-8f27-7cf8d0411f48" containerID="988e62bb18bc28871ad9ad7f8c84dc8f25f413ee81191fe98a9fe172726b8d36" exitCode=0 Sep 11 14:40:58.552192 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.552017 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" Sep 11 14:40:58.552192 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.552045 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" event={"ID":"d9ba867f-8e72-49f6-8f27-7cf8d0411f48","Type":"ContainerDied","Data":"988e62bb18bc28871ad9ad7f8c84dc8f25f413ee81191fe98a9fe172726b8d36"} Sep 11 14:40:58.552192 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.552104 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k" event={"ID":"d9ba867f-8e72-49f6-8f27-7cf8d0411f48","Type":"ContainerDied","Data":"47a5d3c01af85b84c459ad28958d2b87d22a0aa04973a2a9313ff59a852d20df"} Sep 11 14:40:58.552192 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.552135 2573 scope.go:117] "RemoveContainer" containerID="988e62bb18bc28871ad9ad7f8c84dc8f25f413ee81191fe98a9fe172726b8d36" Sep 11 14:40:58.557667 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.557643 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-envoy\") pod \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\" (UID: \"d9ba867f-8e72-49f6-8f27-7cf8d0411f48\") " Sep 11 14:40:58.557912 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.557897 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:58.557979 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.557914 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdvd5\" (UniqueName: \"kubernetes.io/projected/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-kube-api-access-xdvd5\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:58.557979 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.557923 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:58.557979 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.557933 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:58.557979 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.557941 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:58.557979 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.557949 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:58.559573 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.559547 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9ba867f-8e72-49f6-8f27-7cf8d0411f48" (UID: "d9ba867f-8e72-49f6-8f27-7cf8d0411f48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:40:58.562259 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.562242 2573 scope.go:117] "RemoveContainer" containerID="9951135d36f3293355ddebab627b0f928001da54a3be96d2f1708d3fd6090aa2" Sep 11 14:40:58.570779 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.570765 2573 scope.go:117] "RemoveContainer" containerID="988e62bb18bc28871ad9ad7f8c84dc8f25f413ee81191fe98a9fe172726b8d36" Sep 11 14:40:58.571051 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:40:58.571031 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"988e62bb18bc28871ad9ad7f8c84dc8f25f413ee81191fe98a9fe172726b8d36\": container with ID starting with 988e62bb18bc28871ad9ad7f8c84dc8f25f413ee81191fe98a9fe172726b8d36 not found: ID does not exist" containerID="988e62bb18bc28871ad9ad7f8c84dc8f25f413ee81191fe98a9fe172726b8d36" Sep 11 14:40:58.571103 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.571058 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988e62bb18bc28871ad9ad7f8c84dc8f25f413ee81191fe98a9fe172726b8d36"} err="failed to get container status \"988e62bb18bc28871ad9ad7f8c84dc8f25f413ee81191fe98a9fe172726b8d36\": rpc error: code = NotFound desc = could not find container \"988e62bb18bc28871ad9ad7f8c84dc8f25f413ee81191fe98a9fe172726b8d36\": container with ID starting with 988e62bb18bc28871ad9ad7f8c84dc8f25f413ee81191fe98a9fe172726b8d36 not found: ID does not exist" Sep 11 14:40:58.571103 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.571074 2573 scope.go:117] "RemoveContainer" containerID="9951135d36f3293355ddebab627b0f928001da54a3be96d2f1708d3fd6090aa2" Sep 11 14:40:58.571293 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:40:58.571277 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9951135d36f3293355ddebab627b0f928001da54a3be96d2f1708d3fd6090aa2\": container with ID starting with 9951135d36f3293355ddebab627b0f928001da54a3be96d2f1708d3fd6090aa2 not found: ID does not exist" containerID="9951135d36f3293355ddebab627b0f928001da54a3be96d2f1708d3fd6090aa2" Sep 11 14:40:58.571333 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.571297 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9951135d36f3293355ddebab627b0f928001da54a3be96d2f1708d3fd6090aa2"} err="failed to get container status \"9951135d36f3293355ddebab627b0f928001da54a3be96d2f1708d3fd6090aa2\": rpc error: code = NotFound desc = could not find container \"9951135d36f3293355ddebab627b0f928001da54a3be96d2f1708d3fd6090aa2\": container with ID starting with 9951135d36f3293355ddebab627b0f928001da54a3be96d2f1708d3fd6090aa2 not found: ID does not exist" Sep 11 14:40:58.658589 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.658562 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ba867f-8e72-49f6-8f27-7cf8d0411f48-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:40:58.821579 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.821493 2573 patch_prober.go:28] interesting pod/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 11 14:40:58.821579 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.821561 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" podUID="42d9539d-aec8-44aa-a184-16fc04d21c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 11 14:40:58.880495 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.880460 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k"] Sep 11 14:40:58.885402 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:58.885378 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-cd8e4a-openshift-default-8f9d4fccd-4f27k"] Sep 11 14:40:59.824814 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:59.824785 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:59.825190 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:59.825147 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:40:59.825727 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:40:59.825712 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:41:00.188525 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:00.188494 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9ba867f-8e72-49f6-8f27-7cf8d0411f48" path="/var/lib/kubelet/pods/d9ba867f-8e72-49f6-8f27-7cf8d0411f48/volumes" Sep 11 14:41:48.494150 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:48.494062 2573 scope.go:117] "RemoveContainer" containerID="bba1cdd7f990e1aa7b5614e7deb531480a6c52e0fdd3be3fffa5ec746cb24bcd" Sep 11 14:41:52.371383 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.371351 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr"] Sep 11 14:41:52.371870 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.371851 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e9027ad-28a5-44d9-b935-fcf83bee47de" containerName="istio-proxy" Sep 11 14:41:52.371931 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.371871 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e9027ad-28a5-44d9-b935-fcf83bee47de" containerName="istio-proxy" Sep 11 14:41:52.371931 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.371884 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e9027ad-28a5-44d9-b935-fcf83bee47de" containerName="istio-proxy" Sep 11 14:41:52.371931 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.371890 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e9027ad-28a5-44d9-b935-fcf83bee47de" containerName="istio-proxy" Sep 11 14:41:52.371931 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.371897 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ba867f-8e72-49f6-8f27-7cf8d0411f48" containerName="istio-proxy" Sep 11 14:41:52.371931 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.371902 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ba867f-8e72-49f6-8f27-7cf8d0411f48" containerName="istio-proxy" Sep 11 14:41:52.371931 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.371912 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ba867f-8e72-49f6-8f27-7cf8d0411f48" containerName="istio-proxy" Sep 11 14:41:52.371931 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.371917 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ba867f-8e72-49f6-8f27-7cf8d0411f48" containerName="istio-proxy" Sep 11 14:41:52.372160 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.371983 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e9027ad-28a5-44d9-b935-fcf83bee47de" containerName="istio-proxy" Sep 11 14:41:52.372160 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.371993 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9ba867f-8e72-49f6-8f27-7cf8d0411f48" containerName="istio-proxy" Sep 11 14:41:52.375073 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.375055 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.378341 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.378321 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-902e45-openshift-default-dockercfg-9tn9k\"" Sep 11 14:41:52.390799 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.390776 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr"] Sep 11 14:41:52.474546 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.474502 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-workload-socket\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.474862 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.474819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-credential-socket\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.475032 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.475015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-workload-certs\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.475197 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.475183 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-podinfo\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.475350 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.475333 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7003775-6fc8-4520-a7dc-00ae28846cf3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.475511 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.475495 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-token\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.475685 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.475670 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-data\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.475815 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.475801 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg5fk\" (UniqueName: \"kubernetes.io/projected/d7003775-6fc8-4520-a7dc-00ae28846cf3-kube-api-access-kg5fk\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.475994 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.475978 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-envoy\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.576942 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.576904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-workload-socket\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.576942 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.576939 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-credential-socket\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.577186 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.576954 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-workload-certs\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.577186 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.576986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-podinfo\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.577186 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.577036 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7003775-6fc8-4520-a7dc-00ae28846cf3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.577186 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.577068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-token\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.577186 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.577131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-data\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.577463 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.577226 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg5fk\" (UniqueName: \"kubernetes.io/projected/d7003775-6fc8-4520-a7dc-00ae28846cf3-kube-api-access-kg5fk\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.577463 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.577295 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-envoy\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.577463 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.577349 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-credential-socket\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.577463 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.577399 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-workload-socket\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.577778 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.577757 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-data\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.577825 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.577776 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-workload-certs\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.577899 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.577870 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7003775-6fc8-4520-a7dc-00ae28846cf3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.579453 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.579432 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-podinfo\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.579566 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.579549 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-envoy\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.586774 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.586719 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-token\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.589031 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.589011 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg5fk\" (UniqueName: \"kubernetes.io/projected/d7003775-6fc8-4520-a7dc-00ae28846cf3-kube-api-access-kg5fk\") pod \"e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.684868 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.684750 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:52.967776 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.967747 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr"] Sep 11 14:41:52.969250 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:41:52.969224 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7003775_6fc8_4520_a7dc_00ae28846cf3.slice/crio-3bf4a26cf361aef4907eff8d7465850524ef8c73b4768ef2280cc87dbc086a31 WatchSource:0}: Error finding container 3bf4a26cf361aef4907eff8d7465850524ef8c73b4768ef2280cc87dbc086a31: Status 404 returned error can't find the container with id 3bf4a26cf361aef4907eff8d7465850524ef8c73b4768ef2280cc87dbc086a31 Sep 11 14:41:52.971539 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.971507 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:41:52.971611 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.971571 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:41:52.971611 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:52.971600 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:41:53.880752 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:53.880716 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" event={"ID":"d7003775-6fc8-4520-a7dc-00ae28846cf3","Type":"ContainerStarted","Data":"cb70db520920848fa89412a3923452dde63a0daf5f09326fdc9e5a0e33aa3aed"} Sep 11 14:41:53.881177 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:53.880777 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" event={"ID":"d7003775-6fc8-4520-a7dc-00ae28846cf3","Type":"ContainerStarted","Data":"3bf4a26cf361aef4907eff8d7465850524ef8c73b4768ef2280cc87dbc086a31"} Sep 11 14:41:53.908063 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:53.908015 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" podStartSLOduration=1.9079998470000001 podStartE2EDuration="1.907999847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:53.904427527 +0000 UTC m=+1626.241395709" watchObservedRunningTime="2026-09-11 14:41:53.907999847 +0000 UTC m=+1626.244968037" Sep 11 14:41:54.685007 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:54.684978 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:54.690462 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:54.690441 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:54.886399 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:54.886365 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:41:54.887268 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:41:54.887251 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:42:16.020569 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:16.020538 2573 generic.go:356] "Generic (PLEG): container finished" podID="d7003775-6fc8-4520-a7dc-00ae28846cf3" containerID="cb70db520920848fa89412a3923452dde63a0daf5f09326fdc9e5a0e33aa3aed" exitCode=0 Sep 11 14:42:16.020996 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:16.020614 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" event={"ID":"d7003775-6fc8-4520-a7dc-00ae28846cf3","Type":"ContainerDied","Data":"cb70db520920848fa89412a3923452dde63a0daf5f09326fdc9e5a0e33aa3aed"} Sep 11 14:42:16.021041 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:16.021022 2573 scope.go:117] "RemoveContainer" containerID="cb70db520920848fa89412a3923452dde63a0daf5f09326fdc9e5a0e33aa3aed" Sep 11 14:42:16.022091 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:16.022066 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:42:16.022153 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:16.022131 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:42:16.022192 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:16.022161 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 14:42:17.028310 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:17.028276 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" event={"ID":"d7003775-6fc8-4520-a7dc-00ae28846cf3","Type":"ContainerStarted","Data":"1f56eb9bec55380b7b76dc42d518ac8626d91066d89c0627ab9f2b76160e5893"} Sep 11 14:42:17.685541 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:17.685502 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:42:17.685719 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:17.685685 2573 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 11 14:42:17.685769 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:17.685743 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" podUID="d7003775-6fc8-4520-a7dc-00ae28846cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 11 14:42:18.685106 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:18.685068 2573 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 11 14:42:18.685570 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:18.685126 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" podUID="d7003775-6fc8-4520-a7dc-00ae28846cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 11 14:42:19.685798 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:19.685759 2573 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 11 14:42:19.686273 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:19.685818 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" podUID="d7003775-6fc8-4520-a7dc-00ae28846cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 11 14:42:20.686039 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:20.686006 2573 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 11 14:42:20.686407 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:20.686061 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" podUID="d7003775-6fc8-4520-a7dc-00ae28846cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 11 14:42:21.685223 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:21.685185 2573 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 11 14:42:21.685395 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:21.685242 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" podUID="d7003775-6fc8-4520-a7dc-00ae28846cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 11 14:42:22.685191 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:22.685156 2573 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 11 14:42:22.685582 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:22.685219 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" podUID="d7003775-6fc8-4520-a7dc-00ae28846cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 11 14:42:23.689912 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:23.689880 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:42:23.690348 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:23.690172 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:42:23.690958 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:23.690937 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:42:26.881276 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:26.881242 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z"] Sep 11 14:42:26.881774 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:26.881757 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e9027ad-28a5-44d9-b935-fcf83bee47de" containerName="istio-proxy" Sep 11 14:42:26.881872 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:26.881777 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9ba867f-8e72-49f6-8f27-7cf8d0411f48" containerName="istio-proxy" Sep 11 14:42:26.884461 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:26.884435 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z" Sep 11 14:42:26.886224 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:26.886200 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z"] Sep 11 14:42:26.887808 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:26.887778 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:42:26.887808 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:26.887786 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dp8gp\"" Sep 11 14:42:26.925271 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:26.925236 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d749f4-c912-4e2f-9974-b7730a1a1715-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z\" (UID: \"06d749f4-c912-4e2f-9974-b7730a1a1715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z" Sep 11 14:42:27.026296 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:27.026257 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d749f4-c912-4e2f-9974-b7730a1a1715-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z\" (UID: \"06d749f4-c912-4e2f-9974-b7730a1a1715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z" Sep 11 14:42:27.026923 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:27.026902 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d749f4-c912-4e2f-9974-b7730a1a1715-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z\" (UID: \"06d749f4-c912-4e2f-9974-b7730a1a1715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z" Sep 11 14:42:27.196869 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:27.196765 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z" Sep 11 14:42:27.471028 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:27.470998 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z"] Sep 11 14:42:27.472791 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:42:27.472760 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d749f4_c912_4e2f_9974_b7730a1a1715.slice/crio-97ec041a69ad32275a44ba016e403a284a14179353c25116dc5e770e6839ccfa WatchSource:0}: Error finding container 97ec041a69ad32275a44ba016e403a284a14179353c25116dc5e770e6839ccfa: Status 404 returned error can't find the container with id 97ec041a69ad32275a44ba016e403a284a14179353c25116dc5e770e6839ccfa Sep 11 14:42:28.104442 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:28.104408 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z" event={"ID":"06d749f4-c912-4e2f-9974-b7730a1a1715","Type":"ContainerStarted","Data":"38e7a439eba28f18cd4816100721cf96cb139e4251fae0fdf91a9645ea0491ef"} Sep 11 14:42:28.104857 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:28.104457 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z" event={"ID":"06d749f4-c912-4e2f-9974-b7730a1a1715","Type":"ContainerStarted","Data":"97ec041a69ad32275a44ba016e403a284a14179353c25116dc5e770e6839ccfa"} Sep 11 14:42:28.129046 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:28.128997 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z" podStartSLOduration=2.128982922 podStartE2EDuration="2.128982922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:42:28.126753688 +0000 UTC m=+1660.463721891" watchObservedRunningTime="2026-09-11 14:42:28.128982922 +0000 UTC m=+1660.465951113" Sep 11 14:42:29.113165 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:29.113130 2573 generic.go:356] "Generic (PLEG): container finished" podID="06d749f4-c912-4e2f-9974-b7730a1a1715" containerID="38e7a439eba28f18cd4816100721cf96cb139e4251fae0fdf91a9645ea0491ef" exitCode=0 Sep 11 14:42:29.113565 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:29.113224 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z" event={"ID":"06d749f4-c912-4e2f-9974-b7730a1a1715","Type":"ContainerDied","Data":"38e7a439eba28f18cd4816100721cf96cb139e4251fae0fdf91a9645ea0491ef"} Sep 11 14:42:30.257243 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:30.257219 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z" Sep 11 14:42:30.356469 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:30.356436 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d749f4-c912-4e2f-9974-b7730a1a1715-maas-api-service-ca\") pod \"06d749f4-c912-4e2f-9974-b7730a1a1715\" (UID: \"06d749f4-c912-4e2f-9974-b7730a1a1715\") " Sep 11 14:42:30.356785 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:30.356763 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d749f4-c912-4e2f-9974-b7730a1a1715-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "06d749f4-c912-4e2f-9974-b7730a1a1715" (UID: "06d749f4-c912-4e2f-9974-b7730a1a1715"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:42:30.457705 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:30.457629 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06d749f4-c912-4e2f-9974-b7730a1a1715-maas-api-service-ca\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:42:31.125691 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:31.125666 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z" Sep 11 14:42:31.125884 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:31.125659 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z" event={"ID":"06d749f4-c912-4e2f-9974-b7730a1a1715","Type":"ContainerDied","Data":"97ec041a69ad32275a44ba016e403a284a14179353c25116dc5e770e6839ccfa"} Sep 11 14:42:31.125884 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:31.125787 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97ec041a69ad32275a44ba016e403a284a14179353c25116dc5e770e6839ccfa" Sep 11 14:42:38.389965 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:38.389933 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z"] Sep 11 14:42:38.392542 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:38.392517 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-902e45-xss7z"] Sep 11 14:42:40.186848 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:40.186806 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06d749f4-c912-4e2f-9974-b7730a1a1715" path="/var/lib/kubelet/pods/06d749f4-c912-4e2f-9974-b7730a1a1715/volumes" Sep 11 14:42:45.384554 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:45.384519 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr"] Sep 11 14:42:45.385035 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:45.384782 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" podUID="d7003775-6fc8-4520-a7dc-00ae28846cf3" containerName="istio-proxy" containerID="cri-o://1f56eb9bec55380b7b76dc42d518ac8626d91066d89c0627ab9f2b76160e5893" gracePeriod=30 Sep 11 14:42:50.645066 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.645040 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:42:50.759764 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.759683 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-workload-socket\") pod \"d7003775-6fc8-4520-a7dc-00ae28846cf3\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " Sep 11 14:42:50.759764 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.759731 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-envoy\") pod \"d7003775-6fc8-4520-a7dc-00ae28846cf3\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " Sep 11 14:42:50.760020 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.759802 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-data\") pod \"d7003775-6fc8-4520-a7dc-00ae28846cf3\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " Sep 11 14:42:50.760020 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.759829 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg5fk\" (UniqueName: \"kubernetes.io/projected/d7003775-6fc8-4520-a7dc-00ae28846cf3-kube-api-access-kg5fk\") pod \"d7003775-6fc8-4520-a7dc-00ae28846cf3\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " Sep 11 14:42:50.760020 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.759894 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7003775-6fc8-4520-a7dc-00ae28846cf3-istiod-ca-cert\") pod \"d7003775-6fc8-4520-a7dc-00ae28846cf3\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " Sep 11 14:42:50.760020 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.759918 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-podinfo\") pod \"d7003775-6fc8-4520-a7dc-00ae28846cf3\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " Sep 11 14:42:50.760020 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.759952 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-workload-certs\") pod \"d7003775-6fc8-4520-a7dc-00ae28846cf3\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " Sep 11 14:42:50.760020 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.759988 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d7003775-6fc8-4520-a7dc-00ae28846cf3" (UID: "d7003775-6fc8-4520-a7dc-00ae28846cf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:42:50.760365 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.760025 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-token\") pod \"d7003775-6fc8-4520-a7dc-00ae28846cf3\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " Sep 11 14:42:50.760365 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.760061 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-credential-socket\") pod \"d7003775-6fc8-4520-a7dc-00ae28846cf3\" (UID: \"d7003775-6fc8-4520-a7dc-00ae28846cf3\") " Sep 11 14:42:50.760365 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.760105 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-data" (OuterVolumeSpecName: "istio-data") pod "d7003775-6fc8-4520-a7dc-00ae28846cf3" (UID: "d7003775-6fc8-4520-a7dc-00ae28846cf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:42:50.760365 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.760307 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7003775-6fc8-4520-a7dc-00ae28846cf3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d7003775-6fc8-4520-a7dc-00ae28846cf3" (UID: "d7003775-6fc8-4520-a7dc-00ae28846cf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:42:50.760584 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.760400 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d7003775-6fc8-4520-a7dc-00ae28846cf3" (UID: "d7003775-6fc8-4520-a7dc-00ae28846cf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:42:50.760584 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.760536 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d7003775-6fc8-4520-a7dc-00ae28846cf3" (UID: "d7003775-6fc8-4520-a7dc-00ae28846cf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:42:50.760695 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.760584 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:42:50.760695 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.760604 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:42:50.760695 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.760618 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:42:50.760695 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.760632 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7003775-6fc8-4520-a7dc-00ae28846cf3-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:42:50.762060 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.762030 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d7003775-6fc8-4520-a7dc-00ae28846cf3" (UID: "d7003775-6fc8-4520-a7dc-00ae28846cf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:42:50.762179 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.762079 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d7003775-6fc8-4520-a7dc-00ae28846cf3" (UID: "d7003775-6fc8-4520-a7dc-00ae28846cf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:42:50.762179 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.762094 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7003775-6fc8-4520-a7dc-00ae28846cf3-kube-api-access-kg5fk" (OuterVolumeSpecName: "kube-api-access-kg5fk") pod "d7003775-6fc8-4520-a7dc-00ae28846cf3" (UID: "d7003775-6fc8-4520-a7dc-00ae28846cf3"). InnerVolumeSpecName "kube-api-access-kg5fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:42:50.762179 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.762131 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-token" (OuterVolumeSpecName: "istio-token") pod "d7003775-6fc8-4520-a7dc-00ae28846cf3" (UID: "d7003775-6fc8-4520-a7dc-00ae28846cf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:42:50.861287 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.861252 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:42:50.861287 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.861281 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:42:50.861287 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.861294 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:42:50.861502 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.861306 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7003775-6fc8-4520-a7dc-00ae28846cf3-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:42:50.861502 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:50.861318 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg5fk\" (UniqueName: \"kubernetes.io/projected/d7003775-6fc8-4520-a7dc-00ae28846cf3-kube-api-access-kg5fk\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:42:51.249501 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:51.249469 2573 generic.go:356] "Generic (PLEG): container finished" podID="d7003775-6fc8-4520-a7dc-00ae28846cf3" containerID="1f56eb9bec55380b7b76dc42d518ac8626d91066d89c0627ab9f2b76160e5893" exitCode=0 Sep 11 14:42:51.249707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:51.249518 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" Sep 11 14:42:51.249707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:51.249561 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" event={"ID":"d7003775-6fc8-4520-a7dc-00ae28846cf3","Type":"ContainerDied","Data":"1f56eb9bec55380b7b76dc42d518ac8626d91066d89c0627ab9f2b76160e5893"} Sep 11 14:42:51.249707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:51.249625 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr" event={"ID":"d7003775-6fc8-4520-a7dc-00ae28846cf3","Type":"ContainerDied","Data":"3bf4a26cf361aef4907eff8d7465850524ef8c73b4768ef2280cc87dbc086a31"} Sep 11 14:42:51.249707 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:51.249659 2573 scope.go:117] "RemoveContainer" containerID="1f56eb9bec55380b7b76dc42d518ac8626d91066d89c0627ab9f2b76160e5893" Sep 11 14:42:51.263020 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:51.262999 2573 scope.go:117] "RemoveContainer" containerID="cb70db520920848fa89412a3923452dde63a0daf5f09326fdc9e5a0e33aa3aed" Sep 11 14:42:51.271643 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:51.271618 2573 scope.go:117] "RemoveContainer" containerID="1f56eb9bec55380b7b76dc42d518ac8626d91066d89c0627ab9f2b76160e5893" Sep 11 14:42:51.272018 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:42:51.271998 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f56eb9bec55380b7b76dc42d518ac8626d91066d89c0627ab9f2b76160e5893\": container with ID starting with 1f56eb9bec55380b7b76dc42d518ac8626d91066d89c0627ab9f2b76160e5893 not found: ID does not exist" containerID="1f56eb9bec55380b7b76dc42d518ac8626d91066d89c0627ab9f2b76160e5893" Sep 11 14:42:51.272072 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:51.272026 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f56eb9bec55380b7b76dc42d518ac8626d91066d89c0627ab9f2b76160e5893"} err="failed to get container status \"1f56eb9bec55380b7b76dc42d518ac8626d91066d89c0627ab9f2b76160e5893\": rpc error: code = NotFound desc = could not find container \"1f56eb9bec55380b7b76dc42d518ac8626d91066d89c0627ab9f2b76160e5893\": container with ID starting with 1f56eb9bec55380b7b76dc42d518ac8626d91066d89c0627ab9f2b76160e5893 not found: ID does not exist" Sep 11 14:42:51.272072 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:51.272053 2573 scope.go:117] "RemoveContainer" containerID="cb70db520920848fa89412a3923452dde63a0daf5f09326fdc9e5a0e33aa3aed" Sep 11 14:42:51.272299 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:42:51.272282 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb70db520920848fa89412a3923452dde63a0daf5f09326fdc9e5a0e33aa3aed\": container with ID starting with cb70db520920848fa89412a3923452dde63a0daf5f09326fdc9e5a0e33aa3aed not found: ID does not exist" containerID="cb70db520920848fa89412a3923452dde63a0daf5f09326fdc9e5a0e33aa3aed" Sep 11 14:42:51.272349 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:51.272305 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb70db520920848fa89412a3923452dde63a0daf5f09326fdc9e5a0e33aa3aed"} err="failed to get container status \"cb70db520920848fa89412a3923452dde63a0daf5f09326fdc9e5a0e33aa3aed\": rpc error: code = NotFound desc = could not find container \"cb70db520920848fa89412a3923452dde63a0daf5f09326fdc9e5a0e33aa3aed\": container with ID starting with cb70db520920848fa89412a3923452dde63a0daf5f09326fdc9e5a0e33aa3aed not found: ID does not exist" Sep 11 14:42:51.280610 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:51.280587 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr"] Sep 11 14:42:51.284877 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:51.284853 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-902e45-openshift-default-7874b4dc78-qhlfr"] Sep 11 14:42:52.188113 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:42:52.188079 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7003775-6fc8-4520-a7dc-00ae28846cf3" path="/var/lib/kubelet/pods/d7003775-6fc8-4520-a7dc-00ae28846cf3/volumes" Sep 11 14:43:55.526539 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:43:55.526501 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp"] Sep 11 14:43:55.527018 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:43:55.526701 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" podUID="82b3b60b-cb0a-48bd-982b-ff12f647752a" containerName="istio-proxy" containerID="cri-o://791e7787c9029add5f358537f6ac78ced1a3ac7bb967bd059b88230907fc701c" gracePeriod=30 Sep 11 14:44:00.674508 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.674475 2573 generic.go:356] "Generic (PLEG): container finished" podID="82b3b60b-cb0a-48bd-982b-ff12f647752a" containerID="791e7787c9029add5f358537f6ac78ced1a3ac7bb967bd059b88230907fc701c" exitCode=0 Sep 11 14:44:00.674908 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.674551 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" event={"ID":"82b3b60b-cb0a-48bd-982b-ff12f647752a","Type":"ContainerDied","Data":"791e7787c9029add5f358537f6ac78ced1a3ac7bb967bd059b88230907fc701c"} Sep 11 14:44:00.674908 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.674636 2573 scope.go:117] "RemoveContainer" containerID="ef4ac7e17c497702bd2f36845f7a2782b2c6b122a307ae0c2af7f4efbd348544" Sep 11 14:44:00.786640 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.786574 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:44:00.943507 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.943475 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-podinfo\") pod \"82b3b60b-cb0a-48bd-982b-ff12f647752a\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " Sep 11 14:44:00.943672 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.943533 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-credential-socket\") pod \"82b3b60b-cb0a-48bd-982b-ff12f647752a\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " Sep 11 14:44:00.943672 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.943566 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b3b60b-cb0a-48bd-982b-ff12f647752a-istiod-ca-cert\") pod \"82b3b60b-cb0a-48bd-982b-ff12f647752a\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " Sep 11 14:44:00.943672 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.943598 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-workload-socket\") pod \"82b3b60b-cb0a-48bd-982b-ff12f647752a\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " Sep 11 14:44:00.943672 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.943617 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7c76\" (UniqueName: \"kubernetes.io/projected/82b3b60b-cb0a-48bd-982b-ff12f647752a-kube-api-access-l7c76\") pod \"82b3b60b-cb0a-48bd-982b-ff12f647752a\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " Sep 11 14:44:00.943672 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.943642 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-data\") pod \"82b3b60b-cb0a-48bd-982b-ff12f647752a\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " Sep 11 14:44:00.943950 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.943711 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-envoy\") pod \"82b3b60b-cb0a-48bd-982b-ff12f647752a\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " Sep 11 14:44:00.943950 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.943769 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-workload-certs\") pod \"82b3b60b-cb0a-48bd-982b-ff12f647752a\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " Sep 11 14:44:00.943950 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.943805 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-token\") pod \"82b3b60b-cb0a-48bd-982b-ff12f647752a\" (UID: \"82b3b60b-cb0a-48bd-982b-ff12f647752a\") " Sep 11 14:44:00.944103 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.943937 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "82b3b60b-cb0a-48bd-982b-ff12f647752a" (UID: "82b3b60b-cb0a-48bd-982b-ff12f647752a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:00.944103 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.944031 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b3b60b-cb0a-48bd-982b-ff12f647752a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "82b3b60b-cb0a-48bd-982b-ff12f647752a" (UID: "82b3b60b-cb0a-48bd-982b-ff12f647752a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:00.944103 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.943942 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "82b3b60b-cb0a-48bd-982b-ff12f647752a" (UID: "82b3b60b-cb0a-48bd-982b-ff12f647752a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:00.944103 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.944078 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-data" (OuterVolumeSpecName: "istio-data") pod "82b3b60b-cb0a-48bd-982b-ff12f647752a" (UID: "82b3b60b-cb0a-48bd-982b-ff12f647752a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:00.944307 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.944252 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "82b3b60b-cb0a-48bd-982b-ff12f647752a" (UID: "82b3b60b-cb0a-48bd-982b-ff12f647752a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:00.944366 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.944309 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:00.944366 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.944331 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b3b60b-cb0a-48bd-982b-ff12f647752a-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:00.944366 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.944346 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:00.944366 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.944360 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:00.945938 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.945918 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "82b3b60b-cb0a-48bd-982b-ff12f647752a" (UID: "82b3b60b-cb0a-48bd-982b-ff12f647752a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:00.946035 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.945990 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-token" (OuterVolumeSpecName: "istio-token") pod "82b3b60b-cb0a-48bd-982b-ff12f647752a" (UID: "82b3b60b-cb0a-48bd-982b-ff12f647752a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:00.946213 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.946190 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b3b60b-cb0a-48bd-982b-ff12f647752a-kube-api-access-l7c76" (OuterVolumeSpecName: "kube-api-access-l7c76") pod "82b3b60b-cb0a-48bd-982b-ff12f647752a" (UID: "82b3b60b-cb0a-48bd-982b-ff12f647752a"). InnerVolumeSpecName "kube-api-access-l7c76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:00.946291 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:00.946201 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "82b3b60b-cb0a-48bd-982b-ff12f647752a" (UID: "82b3b60b-cb0a-48bd-982b-ff12f647752a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:44:01.045506 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:01.045433 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:01.045506 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:01.045460 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:01.045506 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:01.045470 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7c76\" (UniqueName: \"kubernetes.io/projected/82b3b60b-cb0a-48bd-982b-ff12f647752a-kube-api-access-l7c76\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:01.045506 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:01.045479 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:01.045506 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:01.045488 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b3b60b-cb0a-48bd-982b-ff12f647752a-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:01.682084 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:01.682053 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" Sep 11 14:44:01.682481 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:01.682062 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp" event={"ID":"82b3b60b-cb0a-48bd-982b-ff12f647752a","Type":"ContainerDied","Data":"de2dd5d49dd8efe64b3eb46a50cf4090ba6caf223384a733c6c27dd0caabebd8"} Sep 11 14:44:01.682481 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:01.682180 2573 scope.go:117] "RemoveContainer" containerID="791e7787c9029add5f358537f6ac78ced1a3ac7bb967bd059b88230907fc701c" Sep 11 14:44:01.720630 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:01.720595 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp"] Sep 11 14:44:01.723341 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:01.723312 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a6c3fc-openshift-default-5cdbcf76bb-r7brp"] Sep 11 14:44:02.187633 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:02.187597 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82b3b60b-cb0a-48bd-982b-ff12f647752a" path="/var/lib/kubelet/pods/82b3b60b-cb0a-48bd-982b-ff12f647752a/volumes" Sep 11 14:44:14.731995 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:14.731964 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6"] Sep 11 14:44:14.732409 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:14.732166 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" podUID="42d9539d-aec8-44aa-a184-16fc04d21c90" containerName="istio-proxy" containerID="cri-o://97b85e4fc5cb953a591a6b7bcada760d24a99cc5417ecd709f1ff97b43dbd133" gracePeriod=30 Sep 11 14:44:14.829680 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:14.829636 2573 patch_prober.go:28] interesting pod/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:44:14.829892 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:14.829789 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" podUID="42d9539d-aec8-44aa-a184-16fc04d21c90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:44:19.997531 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:19.997467 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:44:20.021958 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.021923 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-workload-socket\") pod \"42d9539d-aec8-44aa-a184-16fc04d21c90\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " Sep 11 14:44:20.022127 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.021995 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-envoy\") pod \"42d9539d-aec8-44aa-a184-16fc04d21c90\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " Sep 11 14:44:20.022127 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.022029 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4dm5\" (UniqueName: \"kubernetes.io/projected/42d9539d-aec8-44aa-a184-16fc04d21c90-kube-api-access-n4dm5\") pod \"42d9539d-aec8-44aa-a184-16fc04d21c90\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " Sep 11 14:44:20.022127 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.022084 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d9539d-aec8-44aa-a184-16fc04d21c90-istiod-ca-cert\") pod \"42d9539d-aec8-44aa-a184-16fc04d21c90\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " Sep 11 14:44:20.022127 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.022123 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-credential-socket\") pod \"42d9539d-aec8-44aa-a184-16fc04d21c90\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " Sep 11 14:44:20.022346 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.022152 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-podinfo\") pod \"42d9539d-aec8-44aa-a184-16fc04d21c90\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " Sep 11 14:44:20.022346 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.022190 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "42d9539d-aec8-44aa-a184-16fc04d21c90" (UID: "42d9539d-aec8-44aa-a184-16fc04d21c90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:20.022346 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.022215 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-data\") pod \"42d9539d-aec8-44aa-a184-16fc04d21c90\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " Sep 11 14:44:20.022346 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.022275 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-workload-certs\") pod \"42d9539d-aec8-44aa-a184-16fc04d21c90\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " Sep 11 14:44:20.022346 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.022320 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-token\") pod \"42d9539d-aec8-44aa-a184-16fc04d21c90\" (UID: \"42d9539d-aec8-44aa-a184-16fc04d21c90\") " Sep 11 14:44:20.022617 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.022434 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "42d9539d-aec8-44aa-a184-16fc04d21c90" (UID: "42d9539d-aec8-44aa-a184-16fc04d21c90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:20.022988 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.022734 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d9539d-aec8-44aa-a184-16fc04d21c90-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "42d9539d-aec8-44aa-a184-16fc04d21c90" (UID: "42d9539d-aec8-44aa-a184-16fc04d21c90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:20.022988 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.022751 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-data" (OuterVolumeSpecName: "istio-data") pod "42d9539d-aec8-44aa-a184-16fc04d21c90" (UID: "42d9539d-aec8-44aa-a184-16fc04d21c90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:20.022988 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.022801 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "42d9539d-aec8-44aa-a184-16fc04d21c90" (UID: "42d9539d-aec8-44aa-a184-16fc04d21c90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:20.022988 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.022864 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d9539d-aec8-44aa-a184-16fc04d21c90-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:20.022988 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.022895 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:20.022988 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.022909 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:20.022988 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.022924 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:20.024419 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.024391 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "42d9539d-aec8-44aa-a184-16fc04d21c90" (UID: "42d9539d-aec8-44aa-a184-16fc04d21c90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:20.024535 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.024453 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-token" (OuterVolumeSpecName: "istio-token") pod "42d9539d-aec8-44aa-a184-16fc04d21c90" (UID: "42d9539d-aec8-44aa-a184-16fc04d21c90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:20.024690 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.024668 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d9539d-aec8-44aa-a184-16fc04d21c90-kube-api-access-n4dm5" (OuterVolumeSpecName: "kube-api-access-n4dm5") pod "42d9539d-aec8-44aa-a184-16fc04d21c90" (UID: "42d9539d-aec8-44aa-a184-16fc04d21c90"). InnerVolumeSpecName "kube-api-access-n4dm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:20.024690 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.024683 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "42d9539d-aec8-44aa-a184-16fc04d21c90" (UID: "42d9539d-aec8-44aa-a184-16fc04d21c90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:44:20.123563 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.123516 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:20.123563 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.123556 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:20.123563 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.123569 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:20.123792 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.123582 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4dm5\" (UniqueName: \"kubernetes.io/projected/42d9539d-aec8-44aa-a184-16fc04d21c90-kube-api-access-n4dm5\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:20.123792 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.123594 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d9539d-aec8-44aa-a184-16fc04d21c90-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:20.795147 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.795109 2573 generic.go:356] "Generic (PLEG): container finished" podID="42d9539d-aec8-44aa-a184-16fc04d21c90" containerID="97b85e4fc5cb953a591a6b7bcada760d24a99cc5417ecd709f1ff97b43dbd133" exitCode=0 Sep 11 14:44:20.795346 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.795163 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" Sep 11 14:44:20.795346 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.795188 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" event={"ID":"42d9539d-aec8-44aa-a184-16fc04d21c90","Type":"ContainerDied","Data":"97b85e4fc5cb953a591a6b7bcada760d24a99cc5417ecd709f1ff97b43dbd133"} Sep 11 14:44:20.795346 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.795242 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6" event={"ID":"42d9539d-aec8-44aa-a184-16fc04d21c90","Type":"ContainerDied","Data":"8f9e7a2b14e6059c8fb380b93ed625fa9f4ca9d8755362ee63da53ec0678a238"} Sep 11 14:44:20.795346 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.795267 2573 scope.go:117] "RemoveContainer" containerID="97b85e4fc5cb953a591a6b7bcada760d24a99cc5417ecd709f1ff97b43dbd133" Sep 11 14:44:20.804673 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.804657 2573 scope.go:117] "RemoveContainer" containerID="fe89c6a340ec5de2714a8f8dd4fa5f6a028809bb1087f3897f0543b5d097c6b1" Sep 11 14:44:20.813139 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.813120 2573 scope.go:117] "RemoveContainer" containerID="97b85e4fc5cb953a591a6b7bcada760d24a99cc5417ecd709f1ff97b43dbd133" Sep 11 14:44:20.813422 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:44:20.813405 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b85e4fc5cb953a591a6b7bcada760d24a99cc5417ecd709f1ff97b43dbd133\": container with ID starting with 97b85e4fc5cb953a591a6b7bcada760d24a99cc5417ecd709f1ff97b43dbd133 not found: ID does not exist" containerID="97b85e4fc5cb953a591a6b7bcada760d24a99cc5417ecd709f1ff97b43dbd133" Sep 11 14:44:20.813465 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.813431 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b85e4fc5cb953a591a6b7bcada760d24a99cc5417ecd709f1ff97b43dbd133"} err="failed to get container status \"97b85e4fc5cb953a591a6b7bcada760d24a99cc5417ecd709f1ff97b43dbd133\": rpc error: code = NotFound desc = could not find container \"97b85e4fc5cb953a591a6b7bcada760d24a99cc5417ecd709f1ff97b43dbd133\": container with ID starting with 97b85e4fc5cb953a591a6b7bcada760d24a99cc5417ecd709f1ff97b43dbd133 not found: ID does not exist" Sep 11 14:44:20.813465 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.813447 2573 scope.go:117] "RemoveContainer" containerID="fe89c6a340ec5de2714a8f8dd4fa5f6a028809bb1087f3897f0543b5d097c6b1" Sep 11 14:44:20.813692 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:44:20.813674 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe89c6a340ec5de2714a8f8dd4fa5f6a028809bb1087f3897f0543b5d097c6b1\": container with ID starting with fe89c6a340ec5de2714a8f8dd4fa5f6a028809bb1087f3897f0543b5d097c6b1 not found: ID does not exist" containerID="fe89c6a340ec5de2714a8f8dd4fa5f6a028809bb1087f3897f0543b5d097c6b1" Sep 11 14:44:20.813737 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.813698 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe89c6a340ec5de2714a8f8dd4fa5f6a028809bb1087f3897f0543b5d097c6b1"} err="failed to get container status \"fe89c6a340ec5de2714a8f8dd4fa5f6a028809bb1087f3897f0543b5d097c6b1\": rpc error: code = NotFound desc = could not find container \"fe89c6a340ec5de2714a8f8dd4fa5f6a028809bb1087f3897f0543b5d097c6b1\": container with ID starting with fe89c6a340ec5de2714a8f8dd4fa5f6a028809bb1087f3897f0543b5d097c6b1 not found: ID does not exist" Sep 11 14:44:20.825848 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.825811 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6"] Sep 11 14:44:20.829336 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:20.829311 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-625d10-openshift-default-7df68887fb-bm6v6"] Sep 11 14:44:22.187552 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:22.187517 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42d9539d-aec8-44aa-a184-16fc04d21c90" path="/var/lib/kubelet/pods/42d9539d-aec8-44aa-a184-16fc04d21c90/volumes" Sep 11 14:44:34.136640 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:34.136607 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd"] Sep 11 14:44:34.137187 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:34.136810 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" podUID="3ce241e3-382a-4991-b985-4ea068d2458b" containerName="istio-proxy" containerID="cri-o://28b5ff03fb8da65c422aa7ca1f991870ed13616e344e536787051db34a6d0906" gracePeriod=30 Sep 11 14:44:39.392784 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.392756 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:44:39.503457 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.503361 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ce241e3-382a-4991-b985-4ea068d2458b-istiod-ca-cert\") pod \"3ce241e3-382a-4991-b985-4ea068d2458b\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " Sep 11 14:44:39.503457 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.503417 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ce241e3-382a-4991-b985-4ea068d2458b-istio-podinfo\") pod \"3ce241e3-382a-4991-b985-4ea068d2458b\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " Sep 11 14:44:39.503457 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.503445 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce241e3-382a-4991-b985-4ea068d2458b-istio-token\") pod \"3ce241e3-382a-4991-b985-4ea068d2458b\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " Sep 11 14:44:39.503758 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.503496 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-istio-envoy\") pod \"3ce241e3-382a-4991-b985-4ea068d2458b\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " Sep 11 14:44:39.503758 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.503636 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqjr4\" (UniqueName: \"kubernetes.io/projected/3ce241e3-382a-4991-b985-4ea068d2458b-kube-api-access-kqjr4\") pod \"3ce241e3-382a-4991-b985-4ea068d2458b\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " Sep 11 14:44:39.503758 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.503694 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-workload-certs\") pod \"3ce241e3-382a-4991-b985-4ea068d2458b\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " Sep 11 14:44:39.503758 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.503745 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-istio-data\") pod \"3ce241e3-382a-4991-b985-4ea068d2458b\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " Sep 11 14:44:39.504027 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.503780 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-credential-socket\") pod \"3ce241e3-382a-4991-b985-4ea068d2458b\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " Sep 11 14:44:39.504027 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.503779 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce241e3-382a-4991-b985-4ea068d2458b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ce241e3-382a-4991-b985-4ea068d2458b" (UID: "3ce241e3-382a-4991-b985-4ea068d2458b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:39.504027 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.503926 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-workload-socket\") pod \"3ce241e3-382a-4991-b985-4ea068d2458b\" (UID: \"3ce241e3-382a-4991-b985-4ea068d2458b\") " Sep 11 14:44:39.504183 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.504020 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ce241e3-382a-4991-b985-4ea068d2458b" (UID: "3ce241e3-382a-4991-b985-4ea068d2458b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:39.504183 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.504089 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ce241e3-382a-4991-b985-4ea068d2458b" (UID: "3ce241e3-382a-4991-b985-4ea068d2458b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:39.504299 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.504232 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ce241e3-382a-4991-b985-4ea068d2458b" (UID: "3ce241e3-382a-4991-b985-4ea068d2458b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:39.504299 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.504260 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ce241e3-382a-4991-b985-4ea068d2458b" (UID: "3ce241e3-382a-4991-b985-4ea068d2458b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:39.504560 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.504537 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:39.504627 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.504565 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ce241e3-382a-4991-b985-4ea068d2458b-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:39.504627 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.504580 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:39.504627 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.504595 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:39.504627 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.504608 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:39.505780 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.505752 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ce241e3-382a-4991-b985-4ea068d2458b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ce241e3-382a-4991-b985-4ea068d2458b" (UID: "3ce241e3-382a-4991-b985-4ea068d2458b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:44:39.505916 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.505790 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce241e3-382a-4991-b985-4ea068d2458b-kube-api-access-kqjr4" (OuterVolumeSpecName: "kube-api-access-kqjr4") pod "3ce241e3-382a-4991-b985-4ea068d2458b" (UID: "3ce241e3-382a-4991-b985-4ea068d2458b"). InnerVolumeSpecName "kube-api-access-kqjr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:39.505916 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.505846 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce241e3-382a-4991-b985-4ea068d2458b-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ce241e3-382a-4991-b985-4ea068d2458b" (UID: "3ce241e3-382a-4991-b985-4ea068d2458b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:39.505916 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.505884 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ce241e3-382a-4991-b985-4ea068d2458b" (UID: "3ce241e3-382a-4991-b985-4ea068d2458b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:39.605206 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.605172 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ce241e3-382a-4991-b985-4ea068d2458b-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:39.605206 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.605202 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ce241e3-382a-4991-b985-4ea068d2458b-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:39.605206 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.605211 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ce241e3-382a-4991-b985-4ea068d2458b-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:39.605421 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.605222 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqjr4\" (UniqueName: \"kubernetes.io/projected/3ce241e3-382a-4991-b985-4ea068d2458b-kube-api-access-kqjr4\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:39.906337 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.906297 2573 generic.go:356] "Generic (PLEG): container finished" podID="3ce241e3-382a-4991-b985-4ea068d2458b" containerID="28b5ff03fb8da65c422aa7ca1f991870ed13616e344e536787051db34a6d0906" exitCode=0 Sep 11 14:44:39.906513 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.906348 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" Sep 11 14:44:39.906513 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.906376 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" event={"ID":"3ce241e3-382a-4991-b985-4ea068d2458b","Type":"ContainerDied","Data":"28b5ff03fb8da65c422aa7ca1f991870ed13616e344e536787051db34a6d0906"} Sep 11 14:44:39.906513 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.906435 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd" event={"ID":"3ce241e3-382a-4991-b985-4ea068d2458b","Type":"ContainerDied","Data":"56bd048a994200e4721e228ac1bbe821b6bc9a385e17ba9b943ac4591528ec62"} Sep 11 14:44:39.906513 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.906460 2573 scope.go:117] "RemoveContainer" containerID="28b5ff03fb8da65c422aa7ca1f991870ed13616e344e536787051db34a6d0906" Sep 11 14:44:39.918452 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.918433 2573 scope.go:117] "RemoveContainer" containerID="8df9adf426b378ceb6f4dca22913a5fd0b91cfe52069094b559df50f88c2080c" Sep 11 14:44:39.926820 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.926802 2573 scope.go:117] "RemoveContainer" containerID="28b5ff03fb8da65c422aa7ca1f991870ed13616e344e536787051db34a6d0906" Sep 11 14:44:39.927102 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:44:39.927085 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28b5ff03fb8da65c422aa7ca1f991870ed13616e344e536787051db34a6d0906\": container with ID starting with 28b5ff03fb8da65c422aa7ca1f991870ed13616e344e536787051db34a6d0906 not found: ID does not exist" containerID="28b5ff03fb8da65c422aa7ca1f991870ed13616e344e536787051db34a6d0906" Sep 11 14:44:39.927154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.927112 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b5ff03fb8da65c422aa7ca1f991870ed13616e344e536787051db34a6d0906"} err="failed to get container status \"28b5ff03fb8da65c422aa7ca1f991870ed13616e344e536787051db34a6d0906\": rpc error: code = NotFound desc = could not find container \"28b5ff03fb8da65c422aa7ca1f991870ed13616e344e536787051db34a6d0906\": container with ID starting with 28b5ff03fb8da65c422aa7ca1f991870ed13616e344e536787051db34a6d0906 not found: ID does not exist" Sep 11 14:44:39.927154 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.927128 2573 scope.go:117] "RemoveContainer" containerID="8df9adf426b378ceb6f4dca22913a5fd0b91cfe52069094b559df50f88c2080c" Sep 11 14:44:39.927357 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:44:39.927342 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df9adf426b378ceb6f4dca22913a5fd0b91cfe52069094b559df50f88c2080c\": container with ID starting with 8df9adf426b378ceb6f4dca22913a5fd0b91cfe52069094b559df50f88c2080c not found: ID does not exist" containerID="8df9adf426b378ceb6f4dca22913a5fd0b91cfe52069094b559df50f88c2080c" Sep 11 14:44:39.927395 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.927363 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df9adf426b378ceb6f4dca22913a5fd0b91cfe52069094b559df50f88c2080c"} err="failed to get container status \"8df9adf426b378ceb6f4dca22913a5fd0b91cfe52069094b559df50f88c2080c\": rpc error: code = NotFound desc = could not find container \"8df9adf426b378ceb6f4dca22913a5fd0b91cfe52069094b559df50f88c2080c\": container with ID starting with 8df9adf426b378ceb6f4dca22913a5fd0b91cfe52069094b559df50f88c2080c not found: ID does not exist" Sep 11 14:44:39.934022 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.933998 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd"] Sep 11 14:44:39.936500 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:39.936480 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-72e046-openshift-default-7f66c59656-pqghd"] Sep 11 14:44:40.187716 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:40.187639 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ce241e3-382a-4991-b985-4ea068d2458b" path="/var/lib/kubelet/pods/3ce241e3-382a-4991-b985-4ea068d2458b/volumes" Sep 11 14:44:52.707224 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:52.707187 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb"] Sep 11 14:44:52.707634 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:52.707418 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" podUID="73707577-c5a8-410d-b619-6d3cbf8a7724" containerName="istio-proxy" containerID="cri-o://b57ee078c3dffd8a7c4ccc24c3bb96ed03d4cfc157ac63362dcc53bad672f0ae" gracePeriod=30 Sep 11 14:44:54.271063 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:54.271027 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:44:54.271453 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:54.271090 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" podUID="73707577-c5a8-410d-b619-6d3cbf8a7724" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:44:57.974220 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.974154 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:44:57.979450 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.979413 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-workload-socket\") pod \"73707577-c5a8-410d-b619-6d3cbf8a7724\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " Sep 11 14:44:57.979578 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.979514 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-data\") pod \"73707577-c5a8-410d-b619-6d3cbf8a7724\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " Sep 11 14:44:57.979578 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.979544 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73707577-c5a8-410d-b619-6d3cbf8a7724-istiod-ca-cert\") pod \"73707577-c5a8-410d-b619-6d3cbf8a7724\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " Sep 11 14:44:57.979578 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.979573 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-envoy\") pod \"73707577-c5a8-410d-b619-6d3cbf8a7724\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " Sep 11 14:44:57.979746 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.979602 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-credential-socket\") pod \"73707577-c5a8-410d-b619-6d3cbf8a7724\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " Sep 11 14:44:57.979746 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.979627 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-podinfo\") pod \"73707577-c5a8-410d-b619-6d3cbf8a7724\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " Sep 11 14:44:57.979746 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.979665 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-workload-certs\") pod \"73707577-c5a8-410d-b619-6d3cbf8a7724\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " Sep 11 14:44:57.979746 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.979669 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "73707577-c5a8-410d-b619-6d3cbf8a7724" (UID: "73707577-c5a8-410d-b619-6d3cbf8a7724"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:57.979982 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.979777 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6txd7\" (UniqueName: \"kubernetes.io/projected/73707577-c5a8-410d-b619-6d3cbf8a7724-kube-api-access-6txd7\") pod \"73707577-c5a8-410d-b619-6d3cbf8a7724\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " Sep 11 14:44:57.979982 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.979847 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-token\") pod \"73707577-c5a8-410d-b619-6d3cbf8a7724\" (UID: \"73707577-c5a8-410d-b619-6d3cbf8a7724\") " Sep 11 14:44:57.979982 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.979911 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-data" (OuterVolumeSpecName: "istio-data") pod "73707577-c5a8-410d-b619-6d3cbf8a7724" (UID: "73707577-c5a8-410d-b619-6d3cbf8a7724"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:57.979982 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.979949 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "73707577-c5a8-410d-b619-6d3cbf8a7724" (UID: "73707577-c5a8-410d-b619-6d3cbf8a7724"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:57.980191 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.980170 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-workload-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:57.980251 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.980199 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-workload-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:57.980251 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.980215 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-data\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:57.980251 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.980222 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73707577-c5a8-410d-b619-6d3cbf8a7724-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "73707577-c5a8-410d-b619-6d3cbf8a7724" (UID: "73707577-c5a8-410d-b619-6d3cbf8a7724"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:57.980420 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.980286 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "73707577-c5a8-410d-b619-6d3cbf8a7724" (UID: "73707577-c5a8-410d-b619-6d3cbf8a7724"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:57.982888 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.982864 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "73707577-c5a8-410d-b619-6d3cbf8a7724" (UID: "73707577-c5a8-410d-b619-6d3cbf8a7724"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:57.983080 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.983049 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "73707577-c5a8-410d-b619-6d3cbf8a7724" (UID: "73707577-c5a8-410d-b619-6d3cbf8a7724"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:44:57.983159 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.983106 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-token" (OuterVolumeSpecName: "istio-token") pod "73707577-c5a8-410d-b619-6d3cbf8a7724" (UID: "73707577-c5a8-410d-b619-6d3cbf8a7724"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:57.983159 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:57.983134 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73707577-c5a8-410d-b619-6d3cbf8a7724-kube-api-access-6txd7" (OuterVolumeSpecName: "kube-api-access-6txd7") pod "73707577-c5a8-410d-b619-6d3cbf8a7724" (UID: "73707577-c5a8-410d-b619-6d3cbf8a7724"). InnerVolumeSpecName "kube-api-access-6txd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:58.023443 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.023398 2573 generic.go:356] "Generic (PLEG): container finished" podID="73707577-c5a8-410d-b619-6d3cbf8a7724" containerID="b57ee078c3dffd8a7c4ccc24c3bb96ed03d4cfc157ac63362dcc53bad672f0ae" exitCode=0 Sep 11 14:44:58.023615 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.023439 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" event={"ID":"73707577-c5a8-410d-b619-6d3cbf8a7724","Type":"ContainerDied","Data":"b57ee078c3dffd8a7c4ccc24c3bb96ed03d4cfc157ac63362dcc53bad672f0ae"} Sep 11 14:44:58.023615 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.023465 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" Sep 11 14:44:58.023615 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.023537 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb" event={"ID":"73707577-c5a8-410d-b619-6d3cbf8a7724","Type":"ContainerDied","Data":"57deba62532fa61356bf74ed33bf43fd041259dd45bf9a4b1900b2060c8e06ad"} Sep 11 14:44:58.023615 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.023569 2573 scope.go:117] "RemoveContainer" containerID="b57ee078c3dffd8a7c4ccc24c3bb96ed03d4cfc157ac63362dcc53bad672f0ae" Sep 11 14:44:58.036763 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.036740 2573 scope.go:117] "RemoveContainer" containerID="2db4db4952fe59ab0d99ef6b276c505795fcd7d8787d31d882bcd286e7c74fce" Sep 11 14:44:58.047670 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.047646 2573 scope.go:117] "RemoveContainer" containerID="b57ee078c3dffd8a7c4ccc24c3bb96ed03d4cfc157ac63362dcc53bad672f0ae" Sep 11 14:44:58.048047 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:44:58.048019 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b57ee078c3dffd8a7c4ccc24c3bb96ed03d4cfc157ac63362dcc53bad672f0ae\": container with ID starting with b57ee078c3dffd8a7c4ccc24c3bb96ed03d4cfc157ac63362dcc53bad672f0ae not found: ID does not exist" containerID="b57ee078c3dffd8a7c4ccc24c3bb96ed03d4cfc157ac63362dcc53bad672f0ae" Sep 11 14:44:58.048143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.048058 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b57ee078c3dffd8a7c4ccc24c3bb96ed03d4cfc157ac63362dcc53bad672f0ae"} err="failed to get container status \"b57ee078c3dffd8a7c4ccc24c3bb96ed03d4cfc157ac63362dcc53bad672f0ae\": rpc error: code = NotFound desc = could not find container \"b57ee078c3dffd8a7c4ccc24c3bb96ed03d4cfc157ac63362dcc53bad672f0ae\": container with ID starting with b57ee078c3dffd8a7c4ccc24c3bb96ed03d4cfc157ac63362dcc53bad672f0ae not found: ID does not exist" Sep 11 14:44:58.048143 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.048082 2573 scope.go:117] "RemoveContainer" containerID="2db4db4952fe59ab0d99ef6b276c505795fcd7d8787d31d882bcd286e7c74fce" Sep 11 14:44:58.048412 ip-10-0-131-110 kubenswrapper[2573]: E0911 14:44:58.048390 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db4db4952fe59ab0d99ef6b276c505795fcd7d8787d31d882bcd286e7c74fce\": container with ID starting with 2db4db4952fe59ab0d99ef6b276c505795fcd7d8787d31d882bcd286e7c74fce not found: ID does not exist" containerID="2db4db4952fe59ab0d99ef6b276c505795fcd7d8787d31d882bcd286e7c74fce" Sep 11 14:44:58.048467 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.048419 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db4db4952fe59ab0d99ef6b276c505795fcd7d8787d31d882bcd286e7c74fce"} err="failed to get container status \"2db4db4952fe59ab0d99ef6b276c505795fcd7d8787d31d882bcd286e7c74fce\": rpc error: code = NotFound desc = could not find container \"2db4db4952fe59ab0d99ef6b276c505795fcd7d8787d31d882bcd286e7c74fce\": container with ID starting with 2db4db4952fe59ab0d99ef6b276c505795fcd7d8787d31d882bcd286e7c74fce not found: ID does not exist" Sep 11 14:44:58.056880 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.056857 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb"] Sep 11 14:44:58.061461 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.061440 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-344045-openshift-default-854c64846-qqlsb"] Sep 11 14:44:58.081249 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.081212 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6txd7\" (UniqueName: \"kubernetes.io/projected/73707577-c5a8-410d-b619-6d3cbf8a7724-kube-api-access-6txd7\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:58.081249 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.081250 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-token\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:58.081411 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.081266 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73707577-c5a8-410d-b619-6d3cbf8a7724-istiod-ca-cert\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:58.081411 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.081282 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-envoy\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:58.081411 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.081292 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73707577-c5a8-410d-b619-6d3cbf8a7724-credential-socket\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:58.081411 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.081302 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73707577-c5a8-410d-b619-6d3cbf8a7724-istio-podinfo\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:44:58.187495 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:44:58.187465 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73707577-c5a8-410d-b619-6d3cbf8a7724" path="/var/lib/kubelet/pods/73707577-c5a8-410d-b619-6d3cbf8a7724/volumes" Sep 11 14:45:48.570629 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:45:48.570594 2573 scope.go:117] "RemoveContainer" containerID="b8b747ef807e4326afa16f47c51892b6c7fb16a6731f4a05fd79d1e832c66e61" Sep 11 14:48:48.614091 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:48:48.614059 2573 scope.go:117] "RemoveContainer" containerID="38e7a439eba28f18cd4816100721cf96cb139e4251fae0fdf91a9645ea0491ef" Sep 11 14:50:20.538144 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538111 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd"] Sep 11 14:50:20.538642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538490 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42d9539d-aec8-44aa-a184-16fc04d21c90" containerName="istio-proxy" Sep 11 14:50:20.538642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538503 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d9539d-aec8-44aa-a184-16fc04d21c90" containerName="istio-proxy" Sep 11 14:50:20.538642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538515 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73707577-c5a8-410d-b619-6d3cbf8a7724" containerName="istio-proxy" Sep 11 14:50:20.538642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538521 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="73707577-c5a8-410d-b619-6d3cbf8a7724" containerName="istio-proxy" Sep 11 14:50:20.538642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538537 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce241e3-382a-4991-b985-4ea068d2458b" containerName="istio-proxy" Sep 11 14:50:20.538642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538543 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce241e3-382a-4991-b985-4ea068d2458b" containerName="istio-proxy" Sep 11 14:50:20.538642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538550 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73707577-c5a8-410d-b619-6d3cbf8a7724" containerName="istio-proxy" Sep 11 14:50:20.538642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538555 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="73707577-c5a8-410d-b619-6d3cbf8a7724" containerName="istio-proxy" Sep 11 14:50:20.538642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538563 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82b3b60b-cb0a-48bd-982b-ff12f647752a" containerName="istio-proxy" Sep 11 14:50:20.538642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538568 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="82b3b60b-cb0a-48bd-982b-ff12f647752a" containerName="istio-proxy" Sep 11 14:50:20.538642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538573 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d749f4-c912-4e2f-9974-b7730a1a1715" containerName="revoke-keys" Sep 11 14:50:20.538642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538578 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d749f4-c912-4e2f-9974-b7730a1a1715" containerName="revoke-keys" Sep 11 14:50:20.538642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538588 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7003775-6fc8-4520-a7dc-00ae28846cf3" containerName="istio-proxy" Sep 11 14:50:20.538642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538593 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7003775-6fc8-4520-a7dc-00ae28846cf3" containerName="istio-proxy" Sep 11 14:50:20.538642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538602 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7003775-6fc8-4520-a7dc-00ae28846cf3" containerName="istio-proxy" Sep 11 14:50:20.538642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538608 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7003775-6fc8-4520-a7dc-00ae28846cf3" containerName="istio-proxy" Sep 11 14:50:20.539202 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538669 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="82b3b60b-cb0a-48bd-982b-ff12f647752a" containerName="istio-proxy" Sep 11 14:50:20.539202 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538678 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="73707577-c5a8-410d-b619-6d3cbf8a7724" containerName="istio-proxy" Sep 11 14:50:20.539202 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538684 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="42d9539d-aec8-44aa-a184-16fc04d21c90" containerName="istio-proxy" Sep 11 14:50:20.539202 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538690 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7003775-6fc8-4520-a7dc-00ae28846cf3" containerName="istio-proxy" Sep 11 14:50:20.539202 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538695 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="06d749f4-c912-4e2f-9974-b7730a1a1715" containerName="revoke-keys" Sep 11 14:50:20.539202 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538701 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ce241e3-382a-4991-b985-4ea068d2458b" containerName="istio-proxy" Sep 11 14:50:20.539202 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538706 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ce241e3-382a-4991-b985-4ea068d2458b" containerName="istio-proxy" Sep 11 14:50:20.539202 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538711 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="82b3b60b-cb0a-48bd-982b-ff12f647752a" containerName="istio-proxy" Sep 11 14:50:20.539202 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.538717 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="42d9539d-aec8-44aa-a184-16fc04d21c90" containerName="istio-proxy" Sep 11 14:50:20.541974 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.541957 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:20.546202 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.546180 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tvhzt\"" Sep 11 14:50:20.546343 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.546235 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 14:50:20.546343 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.546260 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 14:50:20.546343 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.546275 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:50:20.546343 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.546259 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 14:50:20.559437 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.559415 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd"] Sep 11 14:50:20.684117 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.684080 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-metrics-certs\") pod \"maas-api-6c98db545c-6kvmd\" (UID: \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\") " pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:20.684303 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.684126 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwmhx\" (UniqueName: \"kubernetes.io/projected/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-kube-api-access-xwmhx\") pod \"maas-api-6c98db545c-6kvmd\" (UID: \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\") " pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:20.684303 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.684217 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-openshift-service-ca\") pod \"maas-api-6c98db545c-6kvmd\" (UID: \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\") " pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:20.684303 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.684267 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-maas-api-tls\") pod \"maas-api-6c98db545c-6kvmd\" (UID: \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\") " pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:20.785265 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.785230 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-openshift-service-ca\") pod \"maas-api-6c98db545c-6kvmd\" (UID: \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\") " pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:20.785265 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.785269 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-maas-api-tls\") pod \"maas-api-6c98db545c-6kvmd\" (UID: \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\") " pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:20.785458 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.785354 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-metrics-certs\") pod \"maas-api-6c98db545c-6kvmd\" (UID: \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\") " pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:20.785458 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.785386 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwmhx\" (UniqueName: \"kubernetes.io/projected/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-kube-api-access-xwmhx\") pod \"maas-api-6c98db545c-6kvmd\" (UID: \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\") " pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:20.785952 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.785923 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-openshift-service-ca\") pod \"maas-api-6c98db545c-6kvmd\" (UID: \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\") " pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:20.787869 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.787822 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-metrics-certs\") pod \"maas-api-6c98db545c-6kvmd\" (UID: \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\") " pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:20.788024 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.788003 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-maas-api-tls\") pod \"maas-api-6c98db545c-6kvmd\" (UID: \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\") " pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:20.795579 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.795506 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwmhx\" (UniqueName: \"kubernetes.io/projected/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-kube-api-access-xwmhx\") pod \"maas-api-6c98db545c-6kvmd\" (UID: \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\") " pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:20.853134 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:20.853109 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:21.135756 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:21.135729 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd"] Sep 11 14:50:21.136412 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:50:21.136388 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ebe2ae_bac6_4375_acbd_7b7f6bd03cd4.slice/crio-40360b3375080062a4caa5e93b2101ffb06347621f7d26af761e30f32a0080ef WatchSource:0}: Error finding container 40360b3375080062a4caa5e93b2101ffb06347621f7d26af761e30f32a0080ef: Status 404 returned error can't find the container with id 40360b3375080062a4caa5e93b2101ffb06347621f7d26af761e30f32a0080ef Sep 11 14:50:21.137798 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:21.137781 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:50:21.833361 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:21.833317 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" event={"ID":"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4","Type":"ContainerStarted","Data":"40360b3375080062a4caa5e93b2101ffb06347621f7d26af761e30f32a0080ef"} Sep 11 14:50:23.848569 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:23.848531 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" event={"ID":"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4","Type":"ContainerStarted","Data":"dabb2adc1ba2f53ed039b143f73c32341e046975dbbe470601ce1ee365a8d781"} Sep 11 14:50:23.848952 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:23.848619 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:23.872152 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:23.872066 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" podStartSLOduration=1.4059659070000001 podStartE2EDuration="3.872052634s" totalImagesPullingTime="2.466086727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:50:23.868848882 +0000 UTC m=+2136.205817081" watchObservedRunningTime="2026-09-11 14:50:23.872052634 +0000 UTC m=+2136.209020825" Sep 11 14:50:27.736295 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:27.736263 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd"] Sep 11 14:50:27.736725 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:27.736461 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" podUID="f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4" containerName="maas-api" containerID="cri-o://dabb2adc1ba2f53ed039b143f73c32341e046975dbbe470601ce1ee365a8d781" gracePeriod=30 Sep 11 14:50:27.874246 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:27.874215 2573 generic.go:356] "Generic (PLEG): container finished" podID="f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4" containerID="dabb2adc1ba2f53ed039b143f73c32341e046975dbbe470601ce1ee365a8d781" exitCode=0 Sep 11 14:50:27.874379 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:27.874274 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" event={"ID":"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4","Type":"ContainerDied","Data":"dabb2adc1ba2f53ed039b143f73c32341e046975dbbe470601ce1ee365a8d781"} Sep 11 14:50:27.990916 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:27.990861 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:28.158807 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.158774 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-metrics-certs\") pod \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\" (UID: \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\") " Sep 11 14:50:28.158995 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.158893 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-openshift-service-ca\") pod \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\" (UID: \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\") " Sep 11 14:50:28.158995 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.158935 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwmhx\" (UniqueName: \"kubernetes.io/projected/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-kube-api-access-xwmhx\") pod \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\" (UID: \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\") " Sep 11 14:50:28.159081 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.159016 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-maas-api-tls\") pod \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\" (UID: \"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4\") " Sep 11 14:50:28.159271 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.159235 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4" (UID: "f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:50:28.161130 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.161102 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-kube-api-access-xwmhx" (OuterVolumeSpecName: "kube-api-access-xwmhx") pod "f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4" (UID: "f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4"). InnerVolumeSpecName "kube-api-access-xwmhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:50:28.161223 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.161139 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4" (UID: "f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:50:28.161223 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.161186 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4" (UID: "f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:50:28.260406 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.260336 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-metrics-certs\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:50:28.260406 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.260365 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-openshift-service-ca\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:50:28.260406 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.260376 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwmhx\" (UniqueName: \"kubernetes.io/projected/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-kube-api-access-xwmhx\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:50:28.260406 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.260385 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4-maas-api-tls\") on node \"ip-10-0-131-110.ec2.internal\" DevicePath \"\"" Sep 11 14:50:28.882174 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.882138 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" event={"ID":"f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4","Type":"ContainerDied","Data":"40360b3375080062a4caa5e93b2101ffb06347621f7d26af761e30f32a0080ef"} Sep 11 14:50:28.882615 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.882204 2573 scope.go:117] "RemoveContainer" containerID="dabb2adc1ba2f53ed039b143f73c32341e046975dbbe470601ce1ee365a8d781" Sep 11 14:50:28.882615 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.882149 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd" Sep 11 14:50:28.902982 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.902957 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd"] Sep 11 14:50:28.905310 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:28.905289 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c98db545c-6kvmd"] Sep 11 14:50:30.187495 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:30.187461 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4" path="/var/lib/kubelet/pods/f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4/volumes" Sep 11 14:50:49.308434 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:49.308406 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86cb9f98d8-l4slv_302952be-9e3f-4c1c-95b6-b4d3054776cc/authorino/0.log" Sep 11 14:50:55.449687 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:55.449653 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2lfxz_1c819d4e-3764-4849-aa8d-b6f5b5385124/server/0.log" Sep 11 14:50:55.580232 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:55.580199 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-g9xn5_ff9827c8-e340-4b45-9b2f-45c3d53932ad/manager/1.log" Sep 11 14:50:55.837848 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:55.837813 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74798b5f56-mhrv5_5358a5de-1c21-4cfc-a220-9a45c8e1d934/copy-manifests/0.log" Sep 11 14:50:55.856439 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:55.856415 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74798b5f56-mhrv5_5358a5de-1c21-4cfc-a220-9a45c8e1d934/manager/0.log" Sep 11 14:50:56.584562 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:56.584531 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818950-4j94k_788a148c-256b-4d0b-a54b-e42c6d856a2e/cleanup/0.log" Sep 11 14:50:57.655704 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:57.655675 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86cb9f98d8-l4slv_302952be-9e3f-4c1c-95b6-b4d3054776cc/authorino/0.log" Sep 11 14:50:58.784647 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:58.784611 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8_011dd08d-5c4f-4801-b6ec-dff67cd76120/istio-proxy/0.log" Sep 11 14:50:59.141741 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:59.141710 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f974d944c-jkqks_71beef92-b67b-4ca2-82ec-803a0fb0f65c/kube-auth-proxy/0.log" Sep 11 14:50:59.266167 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:50:59.266138 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d5dd8_7edf42e8-89d8-4376-b88f-0693f77cbd91/istio-proxy/1.log" Sep 11 14:51:00.027173 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:00.027142 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-9qpct_e41c465b-64ea-4726-8b6a-a7247c636199/main/0.log" Sep 11 14:51:00.637827 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:00.637795 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-f4sp9_979f4f9c-7f7f-4f6f-932a-c258b3a09241/main/0.log" Sep 11 14:51:00.757555 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:00.757526 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-gxm4h_04f40182-4fb3-4cfc-a3aa-459d8169c0f4/main/0.log" Sep 11 14:51:06.711961 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.711918 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q5bsg/must-gather-kfp7s"] Sep 11 14:51:06.712518 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.712496 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce241e3-382a-4991-b985-4ea068d2458b" containerName="istio-proxy" Sep 11 14:51:06.712669 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.712520 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce241e3-382a-4991-b985-4ea068d2458b" containerName="istio-proxy" Sep 11 14:51:06.712669 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.712533 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82b3b60b-cb0a-48bd-982b-ff12f647752a" containerName="istio-proxy" Sep 11 14:51:06.712669 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.712543 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="82b3b60b-cb0a-48bd-982b-ff12f647752a" containerName="istio-proxy" Sep 11 14:51:06.712669 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.712576 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4" containerName="maas-api" Sep 11 14:51:06.712669 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.712584 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4" containerName="maas-api" Sep 11 14:51:06.712669 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.712598 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42d9539d-aec8-44aa-a184-16fc04d21c90" containerName="istio-proxy" Sep 11 14:51:06.712669 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.712606 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d9539d-aec8-44aa-a184-16fc04d21c90" containerName="istio-proxy" Sep 11 14:51:06.713041 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.712714 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="73707577-c5a8-410d-b619-6d3cbf8a7724" containerName="istio-proxy" Sep 11 14:51:06.713041 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.712726 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7003775-6fc8-4520-a7dc-00ae28846cf3" containerName="istio-proxy" Sep 11 14:51:06.713041 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.712739 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5ebe2ae-bac6-4375-acbd-7b7f6bd03cd4" containerName="maas-api" Sep 11 14:51:06.719310 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.719294 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5bsg/must-gather-kfp7s" Sep 11 14:51:06.722533 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.722507 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q5bsg\"/\"openshift-service-ca.crt\"" Sep 11 14:51:06.723400 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.723368 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q5bsg\"/\"default-dockercfg-lmmcf\"" Sep 11 14:51:06.723400 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.723396 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q5bsg\"/\"kube-root-ca.crt\"" Sep 11 14:51:06.725544 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.725504 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5bsg/must-gather-kfp7s"] Sep 11 14:51:06.809897 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.809873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8952\" (UniqueName: \"kubernetes.io/projected/3ae92e97-d58f-4e80-9aec-a9d5fda602fe-kube-api-access-n8952\") pod \"must-gather-kfp7s\" (UID: \"3ae92e97-d58f-4e80-9aec-a9d5fda602fe\") " pod="openshift-must-gather-q5bsg/must-gather-kfp7s" Sep 11 14:51:06.810031 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.809930 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3ae92e97-d58f-4e80-9aec-a9d5fda602fe-must-gather-output\") pod \"must-gather-kfp7s\" (UID: \"3ae92e97-d58f-4e80-9aec-a9d5fda602fe\") " pod="openshift-must-gather-q5bsg/must-gather-kfp7s" Sep 11 14:51:06.910662 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.910631 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3ae92e97-d58f-4e80-9aec-a9d5fda602fe-must-gather-output\") pod \"must-gather-kfp7s\" (UID: \"3ae92e97-d58f-4e80-9aec-a9d5fda602fe\") " pod="openshift-must-gather-q5bsg/must-gather-kfp7s" Sep 11 14:51:06.910863 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.910702 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8952\" (UniqueName: \"kubernetes.io/projected/3ae92e97-d58f-4e80-9aec-a9d5fda602fe-kube-api-access-n8952\") pod \"must-gather-kfp7s\" (UID: \"3ae92e97-d58f-4e80-9aec-a9d5fda602fe\") " pod="openshift-must-gather-q5bsg/must-gather-kfp7s" Sep 11 14:51:06.911017 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.910998 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3ae92e97-d58f-4e80-9aec-a9d5fda602fe-must-gather-output\") pod \"must-gather-kfp7s\" (UID: \"3ae92e97-d58f-4e80-9aec-a9d5fda602fe\") " pod="openshift-must-gather-q5bsg/must-gather-kfp7s" Sep 11 14:51:06.920404 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:06.920380 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8952\" (UniqueName: \"kubernetes.io/projected/3ae92e97-d58f-4e80-9aec-a9d5fda602fe-kube-api-access-n8952\") pod \"must-gather-kfp7s\" (UID: \"3ae92e97-d58f-4e80-9aec-a9d5fda602fe\") " pod="openshift-must-gather-q5bsg/must-gather-kfp7s" Sep 11 14:51:07.030198 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:07.030105 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5bsg/must-gather-kfp7s" Sep 11 14:51:07.312381 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:07.312351 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5bsg/must-gather-kfp7s"] Sep 11 14:51:07.313077 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:51:07.313044 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ae92e97_d58f_4e80_9aec_a9d5fda602fe.slice/crio-eb90b30948ff286378986c298940c2c4bdacc00922cf4c061327268b794a3599 WatchSource:0}: Error finding container eb90b30948ff286378986c298940c2c4bdacc00922cf4c061327268b794a3599: Status 404 returned error can't find the container with id eb90b30948ff286378986c298940c2c4bdacc00922cf4c061327268b794a3599 Sep 11 14:51:08.103112 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:08.103078 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5bsg/must-gather-kfp7s" event={"ID":"3ae92e97-d58f-4e80-9aec-a9d5fda602fe","Type":"ContainerStarted","Data":"eb90b30948ff286378986c298940c2c4bdacc00922cf4c061327268b794a3599"} Sep 11 14:51:09.115656 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:09.115604 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5bsg/must-gather-kfp7s" event={"ID":"3ae92e97-d58f-4e80-9aec-a9d5fda602fe","Type":"ContainerStarted","Data":"e8f2d092c7406c693a95d83e75fa002cce23e937c594341eb82bfa7dcf435314"} Sep 11 14:51:09.116130 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:09.115668 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5bsg/must-gather-kfp7s" event={"ID":"3ae92e97-d58f-4e80-9aec-a9d5fda602fe","Type":"ContainerStarted","Data":"ae959d2ebc12b38de07c8fa6a551270582a4cc389ae54cd608aa213dc62ef54b"} Sep 11 14:51:09.137523 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:09.137462 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q5bsg/must-gather-kfp7s" podStartSLOduration=2.342301758 podStartE2EDuration="3.137438992s" totalImagesPullingTime="795.137234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:51:09.134443172 +0000 UTC m=+2181.471411363" watchObservedRunningTime="2026-09-11 14:51:09.137438992 +0000 UTC m=+2181.474407183" Sep 11 14:51:09.926664 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:09.926628 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m8lns_de580cd5-5351-479c-a9ad-0c0936042074/global-pull-secret-syncer/0.log" Sep 11 14:51:10.092962 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:10.092934 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sn4pw_7d87c49f-6095-47cf-8cc2-736f36cb9a74/konnectivity-agent/0.log" Sep 11 14:51:10.182316 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:10.182228 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-110.ec2.internal_ecb7720b7510a32693de89b52bd85266/haproxy/0.log" Sep 11 14:51:14.859509 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:14.859477 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86cb9f98d8-l4slv_302952be-9e3f-4c1c-95b6-b4d3054776cc/authorino/0.log" Sep 11 14:51:16.798918 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:16.798824 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ecc99b9f-1247-41cf-8312-556ccdb5af4a/alertmanager/0.log" Sep 11 14:51:16.831035 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:16.831002 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ecc99b9f-1247-41cf-8312-556ccdb5af4a/config-reloader/0.log" Sep 11 14:51:16.857234 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:16.857161 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ecc99b9f-1247-41cf-8312-556ccdb5af4a/kube-rbac-proxy-web/0.log" Sep 11 14:51:16.883166 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:16.883136 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ecc99b9f-1247-41cf-8312-556ccdb5af4a/kube-rbac-proxy/0.log" Sep 11 14:51:16.906489 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:16.906452 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ecc99b9f-1247-41cf-8312-556ccdb5af4a/kube-rbac-proxy-metric/0.log" Sep 11 14:51:16.930134 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:16.930095 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ecc99b9f-1247-41cf-8312-556ccdb5af4a/prom-label-proxy/0.log" Sep 11 14:51:16.957164 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:16.957069 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ecc99b9f-1247-41cf-8312-556ccdb5af4a/init-config-reloader/0.log" Sep 11 14:51:17.059234 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:17.059156 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-l9ctq_6208183e-4c87-4dd4-85e5-e436d602f0dd/kube-state-metrics/0.log" Sep 11 14:51:17.094816 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:17.094789 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-l9ctq_6208183e-4c87-4dd4-85e5-e436d602f0dd/kube-rbac-proxy-main/0.log" Sep 11 14:51:17.120446 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:17.120339 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-l9ctq_6208183e-4c87-4dd4-85e5-e436d602f0dd/kube-rbac-proxy-self/0.log" Sep 11 14:51:17.166379 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:17.166346 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-546c45558c-vlls4_015081a7-847f-458f-a5de-eb1662c35b6f/metrics-server/0.log" Sep 11 14:51:17.294284 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:17.294257 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-67kmg_89e0aaa5-26fa-424e-8407-bd04226f9172/node-exporter/0.log" Sep 11 14:51:17.339691 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:17.339662 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-67kmg_89e0aaa5-26fa-424e-8407-bd04226f9172/kube-rbac-proxy/0.log" Sep 11 14:51:17.377686 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:17.377650 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-67kmg_89e0aaa5-26fa-424e-8407-bd04226f9172/init-textfile/0.log" Sep 11 14:51:17.967023 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:17.966989 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-jw6px_0bcc91a3-c35b-4498-8a55-c67ee9c5662e/prometheus-operator/0.log" Sep 11 14:51:17.997454 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:17.997421 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-jw6px_0bcc91a3-c35b-4498-8a55-c67ee9c5662e/kube-rbac-proxy/0.log" Sep 11 14:51:18.063664 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:18.063591 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c88d674d4-h52n8_ce41dbc1-7d2f-47a6-b809-55eb41019d6c/telemeter-client/0.log" Sep 11 14:51:18.091849 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:18.091796 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c88d674d4-h52n8_ce41dbc1-7d2f-47a6-b809-55eb41019d6c/reload/0.log" Sep 11 14:51:18.117755 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:18.117725 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c88d674d4-h52n8_ce41dbc1-7d2f-47a6-b809-55eb41019d6c/kube-rbac-proxy/0.log" Sep 11 14:51:18.170461 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:18.170426 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-786c97fb9c-p22rl_b15ffddc-f04a-4ee0-b396-e9a4e3883808/thanos-query/0.log" Sep 11 14:51:18.211953 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:18.211919 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-786c97fb9c-p22rl_b15ffddc-f04a-4ee0-b396-e9a4e3883808/kube-rbac-proxy-web/0.log" Sep 11 14:51:18.234640 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:18.234529 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-786c97fb9c-p22rl_b15ffddc-f04a-4ee0-b396-e9a4e3883808/kube-rbac-proxy/0.log" Sep 11 14:51:18.287565 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:18.287527 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-786c97fb9c-p22rl_b15ffddc-f04a-4ee0-b396-e9a4e3883808/prom-label-proxy/0.log" Sep 11 14:51:18.335273 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:18.335162 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-786c97fb9c-p22rl_b15ffddc-f04a-4ee0-b396-e9a4e3883808/kube-rbac-proxy-rules/0.log" Sep 11 14:51:18.372619 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:18.372576 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-786c97fb9c-p22rl_b15ffddc-f04a-4ee0-b396-e9a4e3883808/kube-rbac-proxy-metrics/0.log" Sep 11 14:51:19.628585 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.628529 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-krm4c_8bd8da52-9957-4092-b43d-d3f75992e3b5/networking-console-plugin/0.log" Sep 11 14:51:19.682412 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.682374 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v"] Sep 11 14:51:19.689516 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.689480 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:19.698887 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.698861 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v"] Sep 11 14:51:19.774858 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.774264 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bc91a105-f73d-4b9b-b0ac-8955357d7109-podres\") pod \"perf-node-gather-daemonset-xfp6v\" (UID: \"bc91a105-f73d-4b9b-b0ac-8955357d7109\") " pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:19.774858 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.774311 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc91a105-f73d-4b9b-b0ac-8955357d7109-lib-modules\") pod \"perf-node-gather-daemonset-xfp6v\" (UID: \"bc91a105-f73d-4b9b-b0ac-8955357d7109\") " pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:19.774858 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.774427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc91a105-f73d-4b9b-b0ac-8955357d7109-sys\") pod \"perf-node-gather-daemonset-xfp6v\" (UID: \"bc91a105-f73d-4b9b-b0ac-8955357d7109\") " pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:19.774858 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.774458 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rv55\" (UniqueName: \"kubernetes.io/projected/bc91a105-f73d-4b9b-b0ac-8955357d7109-kube-api-access-8rv55\") pod \"perf-node-gather-daemonset-xfp6v\" (UID: \"bc91a105-f73d-4b9b-b0ac-8955357d7109\") " pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:19.774858 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.774500 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bc91a105-f73d-4b9b-b0ac-8955357d7109-proc\") pod \"perf-node-gather-daemonset-xfp6v\" (UID: \"bc91a105-f73d-4b9b-b0ac-8955357d7109\") " pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:19.875318 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.875282 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bc91a105-f73d-4b9b-b0ac-8955357d7109-podres\") pod \"perf-node-gather-daemonset-xfp6v\" (UID: \"bc91a105-f73d-4b9b-b0ac-8955357d7109\") " pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:19.875318 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.875318 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc91a105-f73d-4b9b-b0ac-8955357d7109-lib-modules\") pod \"perf-node-gather-daemonset-xfp6v\" (UID: \"bc91a105-f73d-4b9b-b0ac-8955357d7109\") " pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:19.875579 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.875421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc91a105-f73d-4b9b-b0ac-8955357d7109-sys\") pod \"perf-node-gather-daemonset-xfp6v\" (UID: \"bc91a105-f73d-4b9b-b0ac-8955357d7109\") " pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:19.875579 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.875457 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rv55\" (UniqueName: \"kubernetes.io/projected/bc91a105-f73d-4b9b-b0ac-8955357d7109-kube-api-access-8rv55\") pod \"perf-node-gather-daemonset-xfp6v\" (UID: \"bc91a105-f73d-4b9b-b0ac-8955357d7109\") " pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:19.875579 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.875472 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bc91a105-f73d-4b9b-b0ac-8955357d7109-podres\") pod \"perf-node-gather-daemonset-xfp6v\" (UID: \"bc91a105-f73d-4b9b-b0ac-8955357d7109\") " pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:19.875579 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.875497 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bc91a105-f73d-4b9b-b0ac-8955357d7109-proc\") pod \"perf-node-gather-daemonset-xfp6v\" (UID: \"bc91a105-f73d-4b9b-b0ac-8955357d7109\") " pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:19.875579 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.875517 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc91a105-f73d-4b9b-b0ac-8955357d7109-sys\") pod \"perf-node-gather-daemonset-xfp6v\" (UID: \"bc91a105-f73d-4b9b-b0ac-8955357d7109\") " pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:19.875579 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.875530 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc91a105-f73d-4b9b-b0ac-8955357d7109-lib-modules\") pod \"perf-node-gather-daemonset-xfp6v\" (UID: \"bc91a105-f73d-4b9b-b0ac-8955357d7109\") " pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:19.875925 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.875599 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bc91a105-f73d-4b9b-b0ac-8955357d7109-proc\") pod \"perf-node-gather-daemonset-xfp6v\" (UID: \"bc91a105-f73d-4b9b-b0ac-8955357d7109\") " pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:19.884940 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:19.884868 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rv55\" (UniqueName: \"kubernetes.io/projected/bc91a105-f73d-4b9b-b0ac-8955357d7109-kube-api-access-8rv55\") pod \"perf-node-gather-daemonset-xfp6v\" (UID: \"bc91a105-f73d-4b9b-b0ac-8955357d7109\") " pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:20.006739 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:20.006699 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:20.334642 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:20.334611 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v"] Sep 11 14:51:20.335737 ip-10-0-131-110 kubenswrapper[2573]: W0911 14:51:20.335693 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbc91a105_f73d_4b9b_b0ac_8955357d7109.slice/crio-208d7679f80c1e4c9ca31c7def102c8cd8803e5df61b276e61e3d445357ad1b1 WatchSource:0}: Error finding container 208d7679f80c1e4c9ca31c7def102c8cd8803e5df61b276e61e3d445357ad1b1: Status 404 returned error can't find the container with id 208d7679f80c1e4c9ca31c7def102c8cd8803e5df61b276e61e3d445357ad1b1 Sep 11 14:51:21.066689 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:21.066658 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-6qbfq_fcc91ee5-a37f-42dc-a11d-b719dcae387a/download-server/0.log" Sep 11 14:51:21.224039 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:21.223995 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" event={"ID":"bc91a105-f73d-4b9b-b0ac-8955357d7109","Type":"ContainerStarted","Data":"d231ce433b3c788803cb7b43449f4d80e135a14dbf85416e6aa47465d81b9dda"} Sep 11 14:51:21.224228 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:21.224066 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:21.224228 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:21.224099 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" event={"ID":"bc91a105-f73d-4b9b-b0ac-8955357d7109","Type":"ContainerStarted","Data":"208d7679f80c1e4c9ca31c7def102c8cd8803e5df61b276e61e3d445357ad1b1"} Sep 11 14:51:21.247434 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:21.247373 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" podStartSLOduration=2.247354808 podStartE2EDuration="2.247354808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:51:21.245569205 +0000 UTC m=+2193.582537392" watchObservedRunningTime="2026-09-11 14:51:21.247354808 +0000 UTC m=+2193.584322999" Sep 11 14:51:22.787588 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:22.787554 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4gpp4_68438637-9ab0-4bfe-9074-767a01a364b5/dns/0.log" Sep 11 14:51:22.811687 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:22.811661 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4gpp4_68438637-9ab0-4bfe-9074-767a01a364b5/kube-rbac-proxy/0.log" Sep 11 14:51:22.966798 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:22.966768 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l7mvd_ee1912ca-8cb0-4013-bc95-5cb2210bbfe3/dns-node-resolver/0.log" Sep 11 14:51:23.656855 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:23.656808 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pb9cz_9c142fa3-73aa-434a-a3c8-9d3694c03001/node-ca/0.log" Sep 11 14:51:24.688867 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:24.688809 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8_011dd08d-5c4f-4801-b6ec-dff67cd76120/istio-proxy/0.log" Sep 11 14:51:25.039872 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:25.039767 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f974d944c-jkqks_71beef92-b67b-4ca2-82ec-803a0fb0f65c/kube-auth-proxy/0.log" Sep 11 14:51:25.072324 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:25.072286 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d5dd8_7edf42e8-89d8-4376-b88f-0693f77cbd91/istio-proxy/0.log" Sep 11 14:51:25.090852 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:25.090805 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d5dd8_7edf42e8-89d8-4376-b88f-0693f77cbd91/istio-proxy/1.log" Sep 11 14:51:26.143483 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:26.143434 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g5n7k_4f4f4176-30cd-40b6-bc5d-3f1c1d2cd3ff/serve-healthcheck-canary/0.log" Sep 11 14:51:26.978699 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:26.978664 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tvmcz_6eb988f1-4b9c-44db-85c6-5a817446c56d/kube-rbac-proxy/0.log" Sep 11 14:51:27.004259 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:27.004230 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tvmcz_6eb988f1-4b9c-44db-85c6-5a817446c56d/exporter/0.log" Sep 11 14:51:27.031651 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:27.031620 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tvmcz_6eb988f1-4b9c-44db-85c6-5a817446c56d/extractor/0.log" Sep 11 14:51:27.250697 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:27.250597 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q5bsg/perf-node-gather-daemonset-xfp6v" Sep 11 14:51:30.022092 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:30.022058 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2lfxz_1c819d4e-3764-4849-aa8d-b6f5b5385124/server/0.log" Sep 11 14:51:30.131776 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:30.131705 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-g9xn5_ff9827c8-e340-4b45-9b2f-45c3d53932ad/manager/1.log" Sep 11 14:51:30.192847 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:30.192814 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-g9xn5_ff9827c8-e340-4b45-9b2f-45c3d53932ad/manager/0.log" Sep 11 14:51:30.246126 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:30.246098 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8_011dd08d-5c4f-4801-b6ec-dff67cd76120/istio-proxy/0.log" Sep 11 14:51:30.345289 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:30.345251 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74798b5f56-mhrv5_5358a5de-1c21-4cfc-a220-9a45c8e1d934/manager/0.log" Sep 11 14:51:30.369385 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:30.369359 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74798b5f56-mhrv5_5358a5de-1c21-4cfc-a220-9a45c8e1d934/copy-manifests/0.log" Sep 11 14:51:30.557998 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:30.557922 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f974d944c-jkqks_71beef92-b67b-4ca2-82ec-803a0fb0f65c/kube-auth-proxy/0.log" Sep 11 14:51:30.581496 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:30.581467 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d5dd8_7edf42e8-89d8-4376-b88f-0693f77cbd91/istio-proxy/0.log" Sep 11 14:51:30.597030 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:30.597000 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d5dd8_7edf42e8-89d8-4376-b88f-0693f77cbd91/istio-proxy/1.log" Sep 11 14:51:31.854969 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:31.854937 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c4cd778db-mjktf_520952ba-90cd-4bb1-818f-a5aeb4c54de4/manager/1.log" Sep 11 14:51:31.855452 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:31.855431 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c4cd778db-mjktf_520952ba-90cd-4bb1-818f-a5aeb4c54de4/manager/0.log" Sep 11 14:51:32.123530 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:32.123435 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-g2mv8_011dd08d-5c4f-4801-b6ec-dff67cd76120/istio-proxy/0.log" Sep 11 14:51:32.460611 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:32.460541 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f974d944c-jkqks_71beef92-b67b-4ca2-82ec-803a0fb0f65c/kube-auth-proxy/0.log" Sep 11 14:51:32.486951 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:32.486920 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d5dd8_7edf42e8-89d8-4376-b88f-0693f77cbd91/istio-proxy/0.log" Sep 11 14:51:32.507144 ip-10-0-131-110 kubenswrapper[2573]: I0911 14:51:32.507104 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-d5dd8_7edf42e8-89d8-4376-b88f-0693f77cbd91/istio-proxy/1.log"