Sep 11 18:48:14.437885 ip-10-0-143-253 systemd[1]: Starting Kubernetes Kubelet... Sep 11 18:48:14.865991 ip-10-0-143-253 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 18:48:14.865991 ip-10-0-143-253 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 18:48:14.865991 ip-10-0-143-253 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 18:48:14.865991 ip-10-0-143-253 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 18:48:14.865991 ip-10-0-143-253 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 18:48:14.866668 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.866004 2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 18:48:14.869338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869320 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:48:14.869338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869338 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869343 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869348 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869352 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869355 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869358 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869361 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869366 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869369 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869372 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869376 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869378 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869386 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869389 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869392 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869395 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869397 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869400 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869403 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:48:14.869410 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869405 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869408 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869411 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869413 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869416 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869419 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869422 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869425 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869428 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869431 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869433 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869436 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869439 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869441 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869444 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869446 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869449 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869451 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869453 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869456 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:48:14.869889 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869458 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869461 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869464 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869466 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869468 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869471 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869473 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869476 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869478 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869481 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869483 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869486 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869488 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869491 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869494 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869497 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869500 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869502 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869505 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869507 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:48:14.870408 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869509 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869512 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869516 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869519 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869522 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869525 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869528 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869530 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869533 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869536 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869550 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869554 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869556 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869559 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869562 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869564 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869567 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869570 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869573 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:48:14.870916 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869576 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869579 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869581 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869584 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869587 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869589 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869593 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869596 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869599 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869601 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869604 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869607 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869610 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869612 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869615 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869617 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869620 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869623 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869625 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.869628 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:48:14.871368 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870345 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870352 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870355 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870358 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870361 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870365 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870367 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870370 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870373 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870376 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870378 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870381 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870384 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870386 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870389 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870391 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870394 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870397 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870401 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870404 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:48:14.871867 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870406 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870409 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870412 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870414 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870417 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870420 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870422 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870426 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870430 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870433 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870435 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870438 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870440 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870443 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870445 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870448 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870450 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870453 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870460 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:48:14.872362 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870463 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870465 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870468 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870471 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870473 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870476 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870478 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870481 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870483 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870486 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870488 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870492 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870496 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870499 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870501 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870504 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870507 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870509 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870512 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870514 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:48:14.872837 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870517 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870519 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870522 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870524 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870527 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870529 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870532 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870535 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870552 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870557 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870561 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870564 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870568 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870570 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870573 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870576 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870578 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870581 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870583 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870586 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:48:14.873322 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870588 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870591 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870594 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870597 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870600 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870603 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870606 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870609 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870611 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870614 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870616 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870620 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870622 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870625 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870628 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870630 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870634 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870637 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870640 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.870643 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:48:14.873842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871128 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871141 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871145 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871149 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871152 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871155 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871158 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871160 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871163 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871166 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871169 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871172 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871174 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871177 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871180 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871182 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871185 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871189 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871191 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871194 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:48:14.874344 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871197 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871199 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871202 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871205 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871207 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871210 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871213 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871215 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871218 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871221 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871224 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871226 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871229 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871235 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871238 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871241 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871244 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871247 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871250 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871252 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:48:14.874842 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871255 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871258 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871261 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871263 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871266 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871268 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871271 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871274 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871277 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871279 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871282 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871285 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871287 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871290 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871292 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871295 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871298 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871300 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871303 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871305 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:48:14.875404 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871308 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871311 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871313 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871315 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871318 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871322 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871326 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871330 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871332 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871335 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871337 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871340 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871343 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871347 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871351 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871354 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871357 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871360 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:48:14.875947 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871362 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871367 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871372 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871375 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871378 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871382 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871386 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871390 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871392 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871395 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871397 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871400 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871403 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871405 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871408 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871410 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871413 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871416 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871419 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:48:14.876395 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871423 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.871426 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871501 2556 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871554 2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871575 2556 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871579 2556 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871584 2556 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871587 2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871591 2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871595 2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871599 2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871602 2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871605 2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871609 2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871612 2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871615 2556 flags.go:64] FLAG: --cgroup-root="" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871618 2556 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871621 2556 flags.go:64] FLAG: --client-ca-file="" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871624 2556 flags.go:64] FLAG: --cloud-provider="external" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871627 2556 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871633 2556 flags.go:64] FLAG: --cluster-domain="" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871636 2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871639 2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871642 2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 18:48:14.876868 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871645 2556 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871649 2556 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871652 2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871655 2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871659 2556 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871662 2556 flags.go:64] FLAG: --contention-profiling="false" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871665 2556 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871668 2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871673 2556 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871676 2556 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871680 2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871689 2556 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871692 2556 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871695 2556 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871698 2556 flags.go:64] FLAG: --enable-server="true" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871701 2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871709 2556 flags.go:64] FLAG: --event-burst="100" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871712 2556 flags.go:64] FLAG: --event-qps="50" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871715 2556 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871718 2556 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871721 2556 flags.go:64] FLAG: --eviction-hard="" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871725 2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871728 2556 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871731 2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871734 2556 flags.go:64] FLAG: --eviction-soft="" Sep 11 18:48:14.877476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871737 2556 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871740 2556 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871743 2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871746 2556 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871749 2556 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871752 2556 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871755 2556 flags.go:64] FLAG: --feature-gates="" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871758 2556 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871761 2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871764 2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871768 2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871771 2556 flags.go:64] FLAG: --healthz-port="10248" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871774 2556 flags.go:64] FLAG: --help="false" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871777 2556 flags.go:64] FLAG: --hostname-override="ip-10-0-143-253.ec2.internal" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871780 2556 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871785 2556 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871788 2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871791 2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871794 2556 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871804 2556 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871807 2556 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871810 2556 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871813 2556 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 18:48:14.878102 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871816 2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871819 2556 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871822 2556 flags.go:64] FLAG: --kube-reserved="" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871825 2556 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871828 2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871831 2556 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871834 2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871837 2556 flags.go:64] FLAG: --lock-file="" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871840 2556 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871843 2556 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871846 2556 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871851 2556 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871854 2556 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871857 2556 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871860 2556 flags.go:64] FLAG: --logging-format="text" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871863 2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871866 2556 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871869 2556 flags.go:64] FLAG: --manifest-url="" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871872 2556 flags.go:64] FLAG: --manifest-url-header="" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871876 2556 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871880 2556 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871884 2556 flags.go:64] FLAG: --max-pods="110" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871887 2556 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871890 2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871894 2556 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 18:48:14.878701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871897 2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871900 2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871903 2556 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871906 2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871915 2556 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871918 2556 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871922 2556 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871925 2556 flags.go:64] FLAG: --pod-cidr="" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871928 2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871933 2556 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871936 2556 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871939 2556 flags.go:64] FLAG: --pods-per-core="0" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871942 2556 flags.go:64] FLAG: --port="10250" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871945 2556 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871948 2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e21ab274ba44bd50" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871951 2556 flags.go:64] FLAG: --qos-reserved="" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871954 2556 flags.go:64] FLAG: --read-only-port="10255" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871957 2556 flags.go:64] FLAG: --register-node="true" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871960 2556 flags.go:64] FLAG: --register-with-taints="" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871964 2556 flags.go:64] FLAG: --registry-burst="10" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871967 2556 flags.go:64] FLAG: --registry-qps="5" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871969 2556 flags.go:64] FLAG: --reserved-cpus="" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871972 2556 flags.go:64] FLAG: --reserved-memory="" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871976 2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871979 2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 18:48:14.879313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871982 2556 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871985 2556 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871988 2556 flags.go:64] FLAG: --runonce="false" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871991 2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871995 2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.871998 2556 flags.go:64] FLAG: --seccomp-default="false" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872001 2556 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872005 2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872008 2556 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872011 2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872015 2556 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872018 2556 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872021 2556 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872025 2556 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872028 2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872031 2556 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872034 2556 flags.go:64] FLAG: --system-cgroups="" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872037 2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872046 2556 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872049 2556 flags.go:64] FLAG: --tls-cert-file="" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872052 2556 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872060 2556 flags.go:64] FLAG: --tls-min-version="" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872063 2556 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872066 2556 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872069 2556 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 18:48:14.879992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872072 2556 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872075 2556 flags.go:64] FLAG: --v="2" Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872079 2556 flags.go:64] FLAG: --version="false" Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872083 2556 flags.go:64] FLAG: --vmodule="" Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872087 2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.872090 2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872212 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872216 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872219 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872222 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872225 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872229 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872231 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872234 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872237 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872239 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872242 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872244 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872247 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872250 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872258 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:48:14.880610 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872261 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872264 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872266 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872269 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872271 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872274 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872277 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872279 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872282 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872284 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872287 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872289 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872292 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872294 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872297 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872299 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872302 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872305 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872307 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872310 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:48:14.881159 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872312 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872315 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872319 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872322 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872324 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872327 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872329 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872332 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872334 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872337 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872340 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872342 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872350 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872353 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872355 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872358 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872360 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872363 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872365 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:48:14.881661 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872368 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872371 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872373 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872376 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872378 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872381 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872385 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872389 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872391 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872395 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872398 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872407 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872410 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872414 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872416 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872421 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872423 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872428 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872430 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872433 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:48:14.882132 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872448 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872451 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872454 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872456 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872459 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872462 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872470 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872473 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872475 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872478 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872481 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872483 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872486 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872489 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872491 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872494 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872496 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872499 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872501 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872504 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:48:14.882642 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872508 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:48:14.883124 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872511 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:48:14.883124 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872514 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:48:14.883124 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872518 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:48:14.883124 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.872521 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:48:14.883124 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.873214 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 18:48:14.883124 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.882236 2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 18:48:14.883124 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.882255 2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 18:48:14.883124 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.883053 2556 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 18:48:14.883124 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.883062 2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883132 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883138 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883141 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883145 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883148 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883151 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883154 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883157 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883160 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883163 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883166 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883169 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883171 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883174 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883178 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883182 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883185 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883188 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:48:14.883358 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883191 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883194 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883196 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883199 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883202 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883206 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883209 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883212 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883215 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883218 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883221 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883225 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883228 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883231 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883234 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883238 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883241 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883244 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883246 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:48:14.883899 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883249 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883252 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883254 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883257 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883260 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883263 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883265 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883268 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883271 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883274 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883277 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883279 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883282 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883284 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883287 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883289 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883292 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883294 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883297 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:48:14.884384 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883299 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883302 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883305 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883308 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883310 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883313 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883316 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883319 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883322 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883325 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883328 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883330 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883334 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883336 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883339 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883341 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883344 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883347 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883349 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883352 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:48:14.884846 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883354 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883357 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883360 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883362 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883365 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883368 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883371 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883373 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883376 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883378 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883382 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883386 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883388 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883391 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883393 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883396 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883398 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883401 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883404 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883407 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:48:14.885524 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883410 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:48:14.886317 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883413 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:48:14.886317 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883415 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:48:14.886317 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.883420 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 18:48:14.886317 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883576 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 18:48:14.886317 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883581 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 18:48:14.886317 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883584 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 18:48:14.886317 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883587 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 18:48:14.886317 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883590 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 18:48:14.886317 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883592 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 18:48:14.886317 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883595 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 11 18:48:14.886317 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883598 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 18:48:14.886317 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883601 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 18:48:14.886317 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883603 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 18:48:14.886317 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883605 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 18:48:14.886317 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883608 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883610 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883613 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883616 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883618 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883621 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883623 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883626 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883628 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883632 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883636 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883639 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883642 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883645 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883647 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883651 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883654 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883658 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883661 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883664 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 18:48:14.886793 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883666 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883669 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883671 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883674 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883676 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883679 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883681 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883684 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883686 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883689 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883691 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883694 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883696 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883699 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883701 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883704 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883706 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883709 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883711 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883714 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 18:48:14.887292 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883716 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883719 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883721 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883723 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883726 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883729 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883731 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883734 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883736 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883740 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883743 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883745 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883748 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883750 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883753 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883756 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883758 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883761 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883763 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883766 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 18:48:14.887852 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883768 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883771 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883773 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883776 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883779 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883782 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883784 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883787 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883789 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883792 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883794 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883797 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883799 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883801 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883804 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883806 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883809 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883812 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883815 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 18:48:14.888338 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883819 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 18:48:14.888817 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883822 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 18:48:14.888817 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883825 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 18:48:14.888817 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883827 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 18:48:14.888817 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883831 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 18:48:14.888817 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883834 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 18:48:14.888817 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883836 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 18:48:14.888817 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883839 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 18:48:14.888817 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:14.883842 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 18:48:14.888817 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.883846 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 18:48:14.888817 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.883959 2556 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 18:48:14.888817 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.887907 2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 18:48:14.889087 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.888980 2556 server.go:1019] "Starting client certificate rotation" Sep 11 18:48:14.889117 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.889081 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 18:48:14.889145 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.889124 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 18:48:14.891800 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.891786 2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 18:48:14.896047 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.896029 2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 18:48:14.905021 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.905000 2556 log.go:25] "Validated CRI v1 runtime API" Sep 11 18:48:14.910087 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.910072 2556 log.go:25] "Validated CRI v1 image API" Sep 11 18:48:14.910828 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.910812 2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 18:48:14.914147 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.914129 2556 fs.go:135] Filesystem UUIDs: map[24ec4fb6-5464-415f-afbb-ec746abc2d9c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cf95498e-a622-48a4-b2ce-d3f3d9322c66:/dev/nvme0n1p4] Sep 11 18:48:14.914207 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.914147 2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 18:48:14.920182 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.920068 2556 manager.go:216] Machine: {Timestamp:2026-09-11 18:48:14.917712498 +0000 UTC m=+0.371518463 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3136280 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26e535ab58fe2188a8bc31dcc93f6b SystemUUID:ec26e535-ab58-fe21-88a8-bc31dcc93f6b BootID:5af1fcf3-ac92-423a-a225-1db7a14bf799 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:99:5e:ef:6e:a7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:99:5e:ef:6e:a7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:aa:ac:d5:a6:c8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 18:48:14.920182 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.920176 2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 18:48:14.920313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.920301 2556 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 18:48:14.923182 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.923165 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 18:48:14.923964 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.923933 2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 18:48:14.924123 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.923976 2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-253.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 18:48:14.924819 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.924810 2556 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 18:48:14.924860 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.924822 2556 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 18:48:14.924860 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.924830 2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 18:48:14.924860 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.924846 2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 18:48:14.926156 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.926145 2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 18:48:14.930779 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.930768 2556 state_mem.go:36] "Initialized new in-memory state store" Sep 11 18:48:14.930901 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.930892 2556 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 18:48:14.930930 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.930922 2556 kubelet.go:486] "Attempting to sync node with API server" Sep 11 18:48:14.930968 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.930932 2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 18:48:14.930968 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.930944 2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 18:48:14.930968 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.930954 2556 kubelet.go:398] "Adding apiserver pod source" Sep 11 18:48:14.930968 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.930962 2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 18:48:14.931326 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.931315 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 18:48:14.931365 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.931344 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 18:48:14.934057 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.934030 2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 18:48:14.937159 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.937141 2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 18:48:14.937227 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.937174 2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 18:48:14.939172 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.939159 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 18:48:14.939211 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.939176 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 18:48:14.939211 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.939183 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 18:48:14.939211 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.939188 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 18:48:14.939211 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.939196 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 18:48:14.939211 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.939203 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 18:48:14.939346 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.939221 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 18:48:14.939346 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.939227 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 18:48:14.939346 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.939234 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 18:48:14.939346 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.939239 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 18:48:14.939346 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.939248 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 18:48:14.939346 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.939253 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 18:48:14.940189 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.940178 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 18:48:14.940223 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.940191 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 18:48:14.944016 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.943996 2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-253.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 18:48:14.944099 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:14.944026 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 18:48:14.944099 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.944093 2556 server.go:1268] "Started kubelet" Sep 11 18:48:14.944209 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:14.944170 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 18:48:14.944254 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.944178 2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 18:48:14.944254 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.944214 2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 18:48:14.944340 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.944279 2556 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 18:48:14.944921 ip-10-0-143-253 systemd[1]: Started Kubernetes Kubelet. Sep 11 18:48:14.945747 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.945728 2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 18:48:14.947992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.947981 2556 server.go:310] "Adding debug handlers to kubelet server" Sep 11 18:48:14.949312 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:14.949286 2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 18:48:14.951972 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:14.950841 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-253.ec2.internal.18d458b6fe2535e8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-253.ec2.internal,UID:ip-10-0-143-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-253.ec2.internal,},FirstTimestamp:2026-09-11 18:48:14.944073192 +0000 UTC m=+0.397879137,LastTimestamp:2026-09-11 18:48:14.944073192 +0000 UTC m=+0.397879137,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-253.ec2.internal,}" Sep 11 18:48:14.953175 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.953162 2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 18:48:14.953239 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.953175 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 18:48:14.953286 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:14.953239 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-253.ec2.internal\" not found" Sep 11 18:48:14.954049 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.953947 2556 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 18:48:14.954049 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.953963 2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 18:48:14.954049 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.953969 2556 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 18:48:14.954231 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.954140 2556 reconstruct.go:97] "Volume reconstruction finished" Sep 11 18:48:14.954231 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.954149 2556 reconciler.go:29] "Reconciler: start to sync state" Sep 11 18:48:14.954536 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.954521 2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 18:48:14.954640 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.954557 2556 factory.go:55] Registering systemd factory Sep 11 18:48:14.954640 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.954567 2556 factory.go:223] Registration of the systemd container factory successfully Sep 11 18:48:14.954778 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.954767 2556 factory.go:153] Registering CRI-O factory Sep 11 18:48:14.954813 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.954780 2556 factory.go:223] Registration of the crio container factory successfully Sep 11 18:48:14.954813 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.954796 2556 factory.go:103] Registering Raw factory Sep 11 18:48:14.954813 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.954805 2556 manager.go:1196] Started watching for new ooms in manager Sep 11 18:48:14.955210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.955200 2556 manager.go:349] Starting recovery of all containers Sep 11 18:48:14.962201 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:14.962165 2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 18:48:14.962296 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:14.962200 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 18:48:14.969718 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.969689 2556 manager.go:354] Recovery completed Sep 11 18:48:14.972189 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:14.971145 2556 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 11 18:48:14.974151 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.974140 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:48:14.976422 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.976396 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:48:14.976482 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.976441 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:48:14.976482 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.976455 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:48:14.977010 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.976996 2556 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 18:48:14.977010 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.977008 2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 18:48:14.977141 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.977027 2556 state_mem.go:36] "Initialized new in-memory state store" Sep 11 18:48:14.977141 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.977009 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:48:14.977807 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.977790 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:48:14.977899 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.977831 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:48:14.977899 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.977846 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:48:14.978440 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.978422 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2rtlq" Sep 11 18:48:14.978812 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:14.978742 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-253.ec2.internal.18d458b70012d694 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-253.ec2.internal,UID:ip-10-0-143-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-253.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-253.ec2.internal,},FirstTimestamp:2026-09-11 18:48:14.976423572 +0000 UTC m=+0.430229522,LastTimestamp:2026-09-11 18:48:14.976423572 +0000 UTC m=+0.430229522,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-253.ec2.internal,}" Sep 11 18:48:14.979170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.979154 2556 policy_none.go:49] "None policy: Start" Sep 11 18:48:14.979241 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.979174 2556 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 18:48:14.979241 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.979187 2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 18:48:14.981226 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.981212 2556 policy_none.go:47] "Start" Sep 11 18:48:14.985947 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:14.985879 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-253.ec2.internal.18d458b700133263 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-253.ec2.internal,UID:ip-10-0-143-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-253.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-253.ec2.internal,},FirstTimestamp:2026-09-11 18:48:14.976447075 +0000 UTC m=+0.430253022,LastTimestamp:2026-09-11 18:48:14.976447075 +0000 UTC m=+0.430253022,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-253.ec2.internal,}" Sep 11 18:48:14.986552 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:14.986514 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2rtlq" Sep 11 18:48:14.988811 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:14.988724 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-253.ec2.internal.18d458b700136626 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-253.ec2.internal,UID:ip-10-0-143-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-253.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-253.ec2.internal,},FirstTimestamp:2026-09-11 18:48:14.976460326 +0000 UTC m=+0.430266274,LastTimestamp:2026-09-11 18:48:14.976460326 +0000 UTC m=+0.430266274,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-253.ec2.internal,}" Sep 11 18:48:14.998453 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:14.998386 2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-253.ec2.internal.18d458b70012d694\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-253.ec2.internal.18d458b70012d694 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-253.ec2.internal,UID:ip-10-0-143-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-253.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-253.ec2.internal,},FirstTimestamp:2026-09-11 18:48:14.976423572 +0000 UTC m=+0.430229522,LastTimestamp:2026-09-11 18:48:14.977806887 +0000 UTC m=+0.431612841,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-253.ec2.internal,}" Sep 11 18:48:15.004129 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.004114 2556 manager.go:337] "Starting Device Plugin manager" Sep 11 18:48:15.004221 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.004154 2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 18:48:15.004221 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.004165 2556 server.go:87] "Starting device plugin registration server" Sep 11 18:48:15.004504 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.004485 2556 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 18:48:15.004557 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.004506 2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 18:48:15.004660 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.004642 2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 18:48:15.004764 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.004742 2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 18:48:15.004764 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.004752 2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 18:48:15.005279 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.005259 2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 18:48:15.005348 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.005314 2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-253.ec2.internal\" not found" Sep 11 18:48:15.007877 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.007801 2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-253.ec2.internal.18d458b700133263\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-253.ec2.internal.18d458b700133263 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-253.ec2.internal,UID:ip-10-0-143-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-253.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-253.ec2.internal,},FirstTimestamp:2026-09-11 18:48:14.976447075 +0000 UTC m=+0.430253022,LastTimestamp:2026-09-11 18:48:14.977838494 +0000 UTC m=+0.431644441,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-253.ec2.internal,}" Sep 11 18:48:15.034092 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.021418 2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-253.ec2.internal.18d458b700136626\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-253.ec2.internal.18d458b700136626 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-253.ec2.internal,UID:ip-10-0-143-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-253.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-253.ec2.internal,},FirstTimestamp:2026-09-11 18:48:14.976460326 +0000 UTC m=+0.430266274,LastTimestamp:2026-09-11 18:48:14.97785185 +0000 UTC m=+0.431657798,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-253.ec2.internal,}" Sep 11 18:48:15.034092 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.026735 2556 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2591 into the system container \"/system.slice\": " Sep 11 18:48:15.034092 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.033094 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-253.ec2.internal.18d458b702ab2102 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-253.ec2.internal,UID:ip-10-0-143-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-143-253.ec2.internal,},FirstTimestamp:2026-09-11 18:48:15.01995853 +0000 UTC m=+0.473764467,LastTimestamp:2026-09-11 18:48:15.01995853 +0000 UTC m=+0.473764467,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-253.ec2.internal,}" Sep 11 18:48:15.105419 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.105388 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:48:15.106854 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.106832 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:48:15.106967 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.106868 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:48:15.106967 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.106882 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:48:15.106967 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.106916 2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.116097 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.115969 2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-253.ec2.internal.18d458b70012d694\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-253.ec2.internal.18d458b70012d694 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-253.ec2.internal,UID:ip-10-0-143-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-253.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-253.ec2.internal,},FirstTimestamp:2026-09-11 18:48:14.976423572 +0000 UTC m=+0.430229522,LastTimestamp:2026-09-11 18:48:15.106851358 +0000 UTC m=+0.560657306,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-253.ec2.internal,}" Sep 11 18:48:15.121511 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.121416 2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-253.ec2.internal.18d458b700133263\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-253.ec2.internal.18d458b700133263 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-253.ec2.internal,UID:ip-10-0-143-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-253.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-253.ec2.internal,},FirstTimestamp:2026-09-11 18:48:14.976447075 +0000 UTC m=+0.430253022,LastTimestamp:2026-09-11 18:48:15.106874354 +0000 UTC m=+0.560680302,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-253.ec2.internal,}" Sep 11 18:48:15.121617 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.121530 2556 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.127160 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.127074 2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-253.ec2.internal.18d458b700136626\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-253.ec2.internal.18d458b700136626 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-253.ec2.internal,UID:ip-10-0-143-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-253.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-253.ec2.internal,},FirstTimestamp:2026-09-11 18:48:14.976460326 +0000 UTC m=+0.430266274,LastTimestamp:2026-09-11 18:48:15.106887446 +0000 UTC m=+0.560693393,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-253.ec2.internal,}" Sep 11 18:48:15.163626 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.163587 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 18:48:15.164844 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.164826 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 18:48:15.164911 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.164889 2556 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 18:48:15.164956 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.164927 2556 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 18:48:15.165036 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.165023 2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 18:48:15.167424 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.167399 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 18:48:15.168279 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.168263 2556 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-253.ec2.internal\" not found" node="ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.265773 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.265730 2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-253.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal"] Sep 11 18:48:15.265914 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.265817 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:48:15.266806 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.266785 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:48:15.266892 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.266829 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:48:15.266892 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.266843 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:48:15.269128 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.269114 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:48:15.269252 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.269236 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.269310 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.269274 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:48:15.269830 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.269804 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:48:15.269830 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.269833 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:48:15.269986 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.269843 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:48:15.269986 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.269915 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:48:15.269986 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.269938 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:48:15.269986 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.269947 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:48:15.272135 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.272122 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.272203 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.272150 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:48:15.272949 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.272933 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:48:15.273021 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.272968 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:48:15.273021 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.272983 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:48:15.293202 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.293182 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-253.ec2.internal\" not found" node="ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.297407 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.297390 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-253.ec2.internal\" not found" node="ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.321742 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.321713 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:48:15.322649 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.322629 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:48:15.322752 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.322661 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:48:15.322752 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.322671 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:48:15.322752 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.322695 2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.330880 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.330862 2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.330975 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.330891 2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-253.ec2.internal\": node \"ip-10-0-143-253.ec2.internal\" not found" Sep 11 18:48:15.356462 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.356431 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bdd962729f012a9148b1a889b407d7c0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-253.ec2.internal\" (UID: \"bdd962729f012a9148b1a889b407d7c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.356462 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.356461 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/98d66ef951df454b941123496bfb620c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal\" (UID: \"98d66ef951df454b941123496bfb620c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.356644 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.356475 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98d66ef951df454b941123496bfb620c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal\" (UID: \"98d66ef951df454b941123496bfb620c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.365649 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.365622 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-253.ec2.internal\" not found" Sep 11 18:48:15.456680 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.456610 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bdd962729f012a9148b1a889b407d7c0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-253.ec2.internal\" (UID: \"bdd962729f012a9148b1a889b407d7c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.456680 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.456641 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/98d66ef951df454b941123496bfb620c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal\" (UID: \"98d66ef951df454b941123496bfb620c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.456680 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.456661 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98d66ef951df454b941123496bfb620c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal\" (UID: \"98d66ef951df454b941123496bfb620c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.456841 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.456695 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98d66ef951df454b941123496bfb620c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal\" (UID: \"98d66ef951df454b941123496bfb620c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.456841 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.456701 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bdd962729f012a9148b1a889b407d7c0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-253.ec2.internal\" (UID: \"bdd962729f012a9148b1a889b407d7c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.456841 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.456736 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/98d66ef951df454b941123496bfb620c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal\" (UID: \"98d66ef951df454b941123496bfb620c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.466721 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.466695 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-253.ec2.internal\" not found" Sep 11 18:48:15.567727 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.567704 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-253.ec2.internal\" not found" Sep 11 18:48:15.596366 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.596330 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.599803 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.599779 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal" Sep 11 18:48:15.668437 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.668397 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-253.ec2.internal\" not found" Sep 11 18:48:15.769219 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.769145 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-253.ec2.internal\" not found" Sep 11 18:48:15.869727 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.869692 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-253.ec2.internal\" not found" Sep 11 18:48:15.888901 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.888868 2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 18:48:15.889051 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.889033 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 18:48:15.964021 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:15.963986 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 18:48:15.970621 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:15.970595 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-253.ec2.internal\" not found" Sep 11 18:48:16.071181 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:16.071065 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-253.ec2.internal\" not found" Sep 11 18:48:16.127255 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.127198 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 18:43:15 +0000 UTC" deadline="2028-04-22 16:59:06.77785884 +0000 UTC" Sep 11 18:48:16.127255 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.127251 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14134h10m50.650612275s" Sep 11 18:48:16.153989 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.153951 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 18:48:16.156089 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.156067 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientMemory" Sep 11 18:48:16.156199 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.156100 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 18:48:16.156199 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.156110 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeHasSufficientPID" Sep 11 18:48:16.172186 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:16.172164 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-253.ec2.internal\" not found" Sep 11 18:48:16.172903 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.172858 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 18:48:16.183037 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:16.183006 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd962729f012a9148b1a889b407d7c0.slice/crio-66c6bd739bcc8cb56c57171f3d92b278bd56cc1f5bd8f805459dac64373b996d WatchSource:0}: Error finding container 66c6bd739bcc8cb56c57171f3d92b278bd56cc1f5bd8f805459dac64373b996d: Status 404 returned error can't find the container with id 66c6bd739bcc8cb56c57171f3d92b278bd56cc1f5bd8f805459dac64373b996d Sep 11 18:48:16.183276 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:16.183254 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d66ef951df454b941123496bfb620c.slice/crio-05923aec19ecad20ea28073db5f0a96ebaadd9a500ef54d6a7f6ab895a941c73 WatchSource:0}: Error finding container 05923aec19ecad20ea28073db5f0a96ebaadd9a500ef54d6a7f6ab895a941c73: Status 404 returned error can't find the container with id 05923aec19ecad20ea28073db5f0a96ebaadd9a500ef54d6a7f6ab895a941c73 Sep 11 18:48:16.188114 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.188092 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:48:16.258760 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.258725 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 18:48:16.355890 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.355800 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-253.ec2.internal" Sep 11 18:48:16.364611 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.364582 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 18:48:16.365370 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.365357 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal" Sep 11 18:48:16.384491 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.384471 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 18:48:16.932958 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.932928 2556 apiserver.go:52] "Watching apiserver" Sep 11 18:48:16.938779 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.938753 2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 18:48:16.939332 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.939307 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal","openshift-multus/multus-thztx","openshift-multus/network-metrics-daemon-rhmpv","openshift-network-diagnostics/network-check-target-grcgm","kube-system/global-pull-secret-syncer-r2xrg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n","openshift-cluster-node-tuning-operator/tuned-46g48","openshift-multus/multus-additional-cni-plugins-lphqn","openshift-network-operator/iptables-alerter-jtt6k","openshift-ovn-kubernetes/ovnkube-node-42nvj","kube-system/konnectivity-agent-4gnvd","kube-system/kube-apiserver-proxy-ip-10-0-143-253.ec2.internal","openshift-dns/node-resolver-n8gcj","openshift-image-registry/node-ca-vm7pc"] Sep 11 18:48:16.941905 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.941877 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:16.944763 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.944056 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n8gcj" Sep 11 18:48:16.944763 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.944473 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nnv5w\"" Sep 11 18:48:16.944763 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.944601 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:48:16.944763 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.944723 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 18:48:16.946345 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.946323 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 18:48:16.946684 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.946665 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s9nm6\"" Sep 11 18:48:16.946875 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.946859 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 18:48:16.947532 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.947514 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:16.947629 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:16.947598 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:16.949291 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.949271 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:16.949381 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:16.949342 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:16.951026 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.950937 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:16.951026 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:16.950994 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:16.953122 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.953102 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:16.954006 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.953987 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 18:48:16.955306 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.955287 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jtt6k" Sep 11 18:48:16.956110 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.956088 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 18:48:16.956196 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.956131 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 18:48:16.956309 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.956291 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rvf2k\"" Sep 11 18:48:16.956553 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.956097 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 18:48:16.957895 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.957876 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:48:16.958235 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.958218 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 18:48:16.958476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.958461 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9j8fz\"" Sep 11 18:48:16.958754 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.958739 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 18:48:16.959105 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.959087 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vm7pc" Sep 11 18:48:16.961204 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.960599 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-thztx" Sep 11 18:48:16.963095 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.963075 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:16.964012 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.963992 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 18:48:16.964226 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964206 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9krwz\" (UniqueName: \"kubernetes.io/projected/acc4af26-c5e0-47f8-bfaf-28d1927fd13d-kube-api-access-9krwz\") pod \"iptables-alerter-jtt6k\" (UID: \"acc4af26-c5e0-47f8-bfaf-28d1927fd13d\") " pod="openshift-network-operator/iptables-alerter-jtt6k" Sep 11 18:48:16.964301 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964240 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-modprobe-d\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:16.964301 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964262 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-sysctl-conf\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:16.964301 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964286 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-lib-modules\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:16.964449 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964307 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/977bd35b-33a1-4762-bfb1-7645b9dae842-tmp\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:16.964449 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964320 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 18:48:16.964449 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964329 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16a827b3-ee19-4675-ab27-8b47231c2ebc-tmp-dir\") pod \"node-resolver-n8gcj\" (UID: \"16a827b3-ee19-4675-ab27-8b47231c2ebc\") " pod="openshift-dns/node-resolver-n8gcj" Sep 11 18:48:16.964449 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964357 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/953eaca9-ab99-4bce-b95a-ab3d3fbad183-dbus\") pod \"global-pull-secret-syncer-r2xrg\" (UID: \"953eaca9-ab99-4bce-b95a-ab3d3fbad183\") " pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:16.964449 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964374 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret\") pod \"global-pull-secret-syncer-r2xrg\" (UID: \"953eaca9-ab99-4bce-b95a-ab3d3fbad183\") " pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:16.964449 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964390 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:16.964449 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964405 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-sys-fs\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:16.964449 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964426 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgvsw\" (UniqueName: \"kubernetes.io/projected/16a827b3-ee19-4675-ab27-8b47231c2ebc-kube-api-access-jgvsw\") pod \"node-resolver-n8gcj\" (UID: \"16a827b3-ee19-4675-ab27-8b47231c2ebc\") " pod="openshift-dns/node-resolver-n8gcj" Sep 11 18:48:16.964449 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964442 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-registration-dir\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:16.964864 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964458 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-systemd\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:16.964864 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964472 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-run\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:16.964864 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964492 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 18:48:16.964864 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964517 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kml6w\"" Sep 11 18:48:16.964864 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964647 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 18:48:16.964864 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964684 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 18:48:16.964864 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964707 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 18:48:16.964864 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964495 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-sys\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:16.964864 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964426 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5x2tk\"" Sep 11 18:48:16.964864 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964822 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 18:48:16.965420 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.964819 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/953eaca9-ab99-4bce-b95a-ab3d3fbad183-kubelet-config\") pod \"global-pull-secret-syncer-r2xrg\" (UID: \"953eaca9-ab99-4bce-b95a-ab3d3fbad183\") " pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:16.965420 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965084 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6kfj\" (UniqueName: \"kubernetes.io/projected/112256f4-7563-4c95-8204-1f2fee8e0b8a-kube-api-access-j6kfj\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:16.965420 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965110 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-var-lib-kubelet\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:16.965420 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965132 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9lc7\" (UniqueName: \"kubernetes.io/projected/977bd35b-33a1-4762-bfb1-7645b9dae842-kube-api-access-r9lc7\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:16.965420 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965170 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs\") pod \"network-metrics-daemon-rhmpv\" (UID: \"681500e4-89ea-42a9-9d91-0d44e7698747\") " pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:16.965420 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965206 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-socket-dir\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:16.965420 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965245 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acc4af26-c5e0-47f8-bfaf-28d1927fd13d-iptables-alerter-script\") pod \"iptables-alerter-jtt6k\" (UID: \"acc4af26-c5e0-47f8-bfaf-28d1927fd13d\") " pod="openshift-network-operator/iptables-alerter-jtt6k" Sep 11 18:48:16.965420 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965272 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-sysctl-d\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:16.965420 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965307 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-host\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:16.965420 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965320 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 18:48:16.965420 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965337 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acc4af26-c5e0-47f8-bfaf-28d1927fd13d-host-slash\") pod \"iptables-alerter-jtt6k\" (UID: \"acc4af26-c5e0-47f8-bfaf-28d1927fd13d\") " pod="openshift-network-operator/iptables-alerter-jtt6k" Sep 11 18:48:16.965420 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965373 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-sysconfig\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:16.965420 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965404 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-kubernetes\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:16.965913 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965427 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq5wr\" (UniqueName: \"kubernetes.io/projected/681500e4-89ea-42a9-9d91-0d44e7698747-kube-api-access-rq5wr\") pod \"network-metrics-daemon-rhmpv\" (UID: \"681500e4-89ea-42a9-9d91-0d44e7698747\") " pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:16.965913 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965451 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4gnvd" Sep 11 18:48:16.965913 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965464 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtg7v\" (UniqueName: \"kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v\") pod \"network-check-target-grcgm\" (UID: \"fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0\") " pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:16.965913 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965488 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-device-dir\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:16.965913 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965511 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:16.965913 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965587 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-tuned\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:16.965913 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965632 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16a827b3-ee19-4675-ab27-8b47231c2ebc-hosts-file\") pod \"node-resolver-n8gcj\" (UID: \"16a827b3-ee19-4675-ab27-8b47231c2ebc\") " pod="openshift-dns/node-resolver-n8gcj" Sep 11 18:48:16.965913 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965711 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 18:48:16.965913 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965903 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lw4mw\"" Sep 11 18:48:16.966281 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.965931 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 18:48:16.966323 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.966297 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 18:48:16.971651 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.970009 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:16.971651 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.970719 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 18:48:16.971651 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.970773 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 18:48:16.971651 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.971112 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 18:48:16.971651 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.971127 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 18:48:16.971651 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.971380 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mgfth\"" Sep 11 18:48:16.971976 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.971862 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 18:48:16.975238 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.974998 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 18:48:16.975238 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.975226 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 18:48:16.975639 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.975535 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 18:48:16.976607 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:16.976589 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rx5lr\"" Sep 11 18:48:17.022828 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.022804 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9bgdx" Sep 11 18:48:17.036466 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.036445 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9bgdx" Sep 11 18:48:17.054741 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.054719 2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 18:48:17.066206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.065874 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-sysconfig\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.066206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.065919 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq5wr\" (UniqueName: \"kubernetes.io/projected/681500e4-89ea-42a9-9d91-0d44e7698747-kube-api-access-rq5wr\") pod \"network-metrics-daemon-rhmpv\" (UID: \"681500e4-89ea-42a9-9d91-0d44e7698747\") " pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:17.066206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.065949 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-hostroot\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.066206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.065965 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-device-dir\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:17.066206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.065981 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16a827b3-ee19-4675-ab27-8b47231c2ebc-hosts-file\") pod \"node-resolver-n8gcj\" (UID: \"16a827b3-ee19-4675-ab27-8b47231c2ebc\") " pod="openshift-dns/node-resolver-n8gcj" Sep 11 18:48:17.066206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.065991 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-sysconfig\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.066206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066010 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-log-socket\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.066206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066053 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad502b23-3af5-4907-8f22-1c3c619555ca-konnectivity-ca\") pod \"konnectivity-agent-4gnvd\" (UID: \"ad502b23-3af5-4907-8f22-1c3c619555ca\") " pod="kube-system/konnectivity-agent-4gnvd" Sep 11 18:48:17.066206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066076 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16a827b3-ee19-4675-ab27-8b47231c2ebc-hosts-file\") pod \"node-resolver-n8gcj\" (UID: \"16a827b3-ee19-4675-ab27-8b47231c2ebc\") " pod="openshift-dns/node-resolver-n8gcj" Sep 11 18:48:17.066206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066092 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-modprobe-d\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.066206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066117 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16a827b3-ee19-4675-ab27-8b47231c2ebc-tmp-dir\") pod \"node-resolver-n8gcj\" (UID: \"16a827b3-ee19-4675-ab27-8b47231c2ebc\") " pod="openshift-dns/node-resolver-n8gcj" Sep 11 18:48:17.066206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066136 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-run-netns\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.066206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066148 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-device-dir\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:17.066206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066155 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-systemd-units\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.066206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066181 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-modprobe-d\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.066206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066180 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70e064d0-8173-4c6e-b32f-a903dbad84cc-ovnkube-script-lib\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066219 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/953eaca9-ab99-4bce-b95a-ab3d3fbad183-dbus\") pod \"global-pull-secret-syncer-r2xrg\" (UID: \"953eaca9-ab99-4bce-b95a-ab3d3fbad183\") " pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066254 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgvsw\" (UniqueName: \"kubernetes.io/projected/16a827b3-ee19-4675-ab27-8b47231c2ebc-kube-api-access-jgvsw\") pod \"node-resolver-n8gcj\" (UID: \"16a827b3-ee19-4675-ab27-8b47231c2ebc\") " pod="openshift-dns/node-resolver-n8gcj" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066276 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066300 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-registration-dir\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066321 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-run\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066336 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/953eaca9-ab99-4bce-b95a-ab3d3fbad183-dbus\") pod \"global-pull-secret-syncer-r2xrg\" (UID: \"953eaca9-ab99-4bce-b95a-ab3d3fbad183\") " pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066342 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-sys\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066380 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-cnibin\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066393 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-registration-dir\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066403 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-etc-openvswitch\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066428 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fb6ab28-29cd-45d4-8057-68bc5875c32e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066437 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16a827b3-ee19-4675-ab27-8b47231c2ebc-tmp-dir\") pod \"node-resolver-n8gcj\" (UID: \"16a827b3-ee19-4675-ab27-8b47231c2ebc\") " pod="openshift-dns/node-resolver-n8gcj" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066452 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-socket-dir\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066453 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-sys\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066476 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acc4af26-c5e0-47f8-bfaf-28d1927fd13d-iptables-alerter-script\") pod \"iptables-alerter-jtt6k\" (UID: \"acc4af26-c5e0-47f8-bfaf-28d1927fd13d\") " pod="openshift-network-operator/iptables-alerter-jtt6k" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066512 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-run\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.067170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066565 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/864dee94-a189-44d1-a4ec-00fa82d5d67e-cni-binary-copy\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066590 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-run-systemd\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066619 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-cni-bin\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066643 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066669 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fb6ab28-29cd-45d4-8057-68bc5875c32e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066681 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-socket-dir\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066693 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fb6ab28-29cd-45d4-8057-68bc5875c32e-cnibin\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066732 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-host\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066765 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-host\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066771 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acc4af26-c5e0-47f8-bfaf-28d1927fd13d-host-slash\") pod \"iptables-alerter-jtt6k\" (UID: \"acc4af26-c5e0-47f8-bfaf-28d1927fd13d\") " pod="openshift-network-operator/iptables-alerter-jtt6k" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066801 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-kubernetes\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066828 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtg7v\" (UniqueName: \"kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v\") pod \"network-check-target-grcgm\" (UID: \"fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0\") " pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066850 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066873 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-tuned\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066897 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/908b068d-8d94-4fcb-8490-eb8b1ac59a90-host\") pod \"node-ca-vm7pc\" (UID: \"908b068d-8d94-4fcb-8490-eb8b1ac59a90\") " pod="openshift-image-registry/node-ca-vm7pc" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066914 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-etc-selinux\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066922 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-etc-kubernetes\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.067785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066945 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-cni-netd\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066970 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70e064d0-8173-4c6e-b32f-a903dbad84cc-env-overrides\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066994 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70e064d0-8173-4c6e-b32f-a903dbad84cc-ovn-node-metrics-cert\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067019 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9krwz\" (UniqueName: \"kubernetes.io/projected/acc4af26-c5e0-47f8-bfaf-28d1927fd13d-kube-api-access-9krwz\") pod \"iptables-alerter-jtt6k\" (UID: \"acc4af26-c5e0-47f8-bfaf-28d1927fd13d\") " pod="openshift-network-operator/iptables-alerter-jtt6k" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067044 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-sysctl-conf\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067068 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-lib-modules\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066803 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acc4af26-c5e0-47f8-bfaf-28d1927fd13d-host-slash\") pod \"iptables-alerter-jtt6k\" (UID: \"acc4af26-c5e0-47f8-bfaf-28d1927fd13d\") " pod="openshift-network-operator/iptables-alerter-jtt6k" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067087 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/977bd35b-33a1-4762-bfb1-7645b9dae842-tmp\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067108 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-system-cni-dir\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.066874 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-kubernetes\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067131 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-var-lib-cni-bin\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067154 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-run-netns\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067157 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acc4af26-c5e0-47f8-bfaf-28d1927fd13d-iptables-alerter-script\") pod \"iptables-alerter-jtt6k\" (UID: \"acc4af26-c5e0-47f8-bfaf-28d1927fd13d\") " pod="openshift-network-operator/iptables-alerter-jtt6k" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067176 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-run-ovn\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067202 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret\") pod \"global-pull-secret-syncer-r2xrg\" (UID: \"953eaca9-ab99-4bce-b95a-ab3d3fbad183\") " pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067229 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067235 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-lib-modules\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.068605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067254 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-sys-fs\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067264 2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067278 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khnc2\" (UniqueName: \"kubernetes.io/projected/908b068d-8d94-4fcb-8490-eb8b1ac59a90-kube-api-access-khnc2\") pod \"node-ca-vm7pc\" (UID: \"908b068d-8d94-4fcb-8490-eb8b1ac59a90\") " pod="openshift-image-registry/node-ca-vm7pc" Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067303 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-multus-cni-dir\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067342 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-multus-socket-dir-parent\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067370 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-sysctl-conf\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067374 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-run-k8s-cni-cncf-io\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067468 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-sys-fs\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067475 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/112256f4-7563-4c95-8204-1f2fee8e0b8a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067516 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-var-lib-kubelet\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:17.067630 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067641 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-systemd\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:17.067716 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret podName:953eaca9-ab99-4bce-b95a-ab3d3fbad183 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:17.567689479 +0000 UTC m=+3.021495412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret") pod "global-pull-secret-syncer-r2xrg" (UID: "953eaca9-ab99-4bce-b95a-ab3d3fbad183") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067747 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/908b068d-8d94-4fcb-8490-eb8b1ac59a90-serviceca\") pod \"node-ca-vm7pc\" (UID: \"908b068d-8d94-4fcb-8490-eb8b1ac59a90\") " pod="openshift-image-registry/node-ca-vm7pc" Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067776 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/864dee94-a189-44d1-a4ec-00fa82d5d67e-multus-daemon-config\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067817 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-multus-conf-dir\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067841 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-run-multus-certs\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.069399 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067867 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkdtt\" (UniqueName: \"kubernetes.io/projected/864dee94-a189-44d1-a4ec-00fa82d5d67e-kube-api-access-lkdtt\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.070210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067881 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-systemd\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.070210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067919 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-run-openvswitch\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.070210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067948 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fb6ab28-29cd-45d4-8057-68bc5875c32e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.070210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.067982 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/953eaca9-ab99-4bce-b95a-ab3d3fbad183-kubelet-config\") pod \"global-pull-secret-syncer-r2xrg\" (UID: \"953eaca9-ab99-4bce-b95a-ab3d3fbad183\") " pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:17.070210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068009 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6kfj\" (UniqueName: \"kubernetes.io/projected/112256f4-7563-4c95-8204-1f2fee8e0b8a-kube-api-access-j6kfj\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:17.070210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068037 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-var-lib-kubelet\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.070210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068052 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/953eaca9-ab99-4bce-b95a-ab3d3fbad183-kubelet-config\") pod \"global-pull-secret-syncer-r2xrg\" (UID: \"953eaca9-ab99-4bce-b95a-ab3d3fbad183\") " pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:17.070210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068075 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9lc7\" (UniqueName: \"kubernetes.io/projected/977bd35b-33a1-4762-bfb1-7645b9dae842-kube-api-access-r9lc7\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.070210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068098 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-var-lib-kubelet\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.070210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068115 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs\") pod \"network-metrics-daemon-rhmpv\" (UID: \"681500e4-89ea-42a9-9d91-0d44e7698747\") " pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:17.070210 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:17.068167 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:48:17.070210 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:17.068213 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs podName:681500e4-89ea-42a9-9d91-0d44e7698747 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:17.568198675 +0000 UTC m=+3.022004612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs") pod "network-metrics-daemon-rhmpv" (UID: "681500e4-89ea-42a9-9d91-0d44e7698747") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:48:17.070210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068289 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70e064d0-8173-4c6e-b32f-a903dbad84cc-ovnkube-config\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.070210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068322 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfhjl\" (UniqueName: \"kubernetes.io/projected/70e064d0-8173-4c6e-b32f-a903dbad84cc-kube-api-access-qfhjl\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.070210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068343 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad502b23-3af5-4907-8f22-1c3c619555ca-agent-certs\") pod \"konnectivity-agent-4gnvd\" (UID: \"ad502b23-3af5-4907-8f22-1c3c619555ca\") " pod="kube-system/konnectivity-agent-4gnvd" Sep 11 18:48:17.070210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068380 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-slash\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.070988 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068401 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-var-lib-openvswitch\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.070988 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068417 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fb6ab28-29cd-45d4-8057-68bc5875c32e-system-cni-dir\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.070988 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068448 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fb6ab28-29cd-45d4-8057-68bc5875c32e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.070988 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068478 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8qlk\" (UniqueName: \"kubernetes.io/projected/2fb6ab28-29cd-45d4-8057-68bc5875c32e-kube-api-access-x8qlk\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.070988 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068510 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-sysctl-d\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.070988 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068531 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-var-lib-cni-multus\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.070988 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068568 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-kubelet\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.070988 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068593 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-node-log\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.070988 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.068640 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-sysctl-d\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.071402 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.071008 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/977bd35b-33a1-4762-bfb1-7645b9dae842-etc-tuned\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.071492 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.071474 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/977bd35b-33a1-4762-bfb1-7645b9dae842-tmp\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.074306 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:17.074261 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:48:17.074306 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:17.074284 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:48:17.074306 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:17.074298 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mtg7v for pod openshift-network-diagnostics/network-check-target-grcgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:48:17.074522 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:17.074358 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v podName:fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:17.574342207 +0000 UTC m=+3.028148162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mtg7v" (UniqueName: "kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v") pod "network-check-target-grcgm" (UID: "fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:48:17.075698 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.075594 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq5wr\" (UniqueName: \"kubernetes.io/projected/681500e4-89ea-42a9-9d91-0d44e7698747-kube-api-access-rq5wr\") pod \"network-metrics-daemon-rhmpv\" (UID: \"681500e4-89ea-42a9-9d91-0d44e7698747\") " pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:17.075919 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.075877 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9krwz\" (UniqueName: \"kubernetes.io/projected/acc4af26-c5e0-47f8-bfaf-28d1927fd13d-kube-api-access-9krwz\") pod \"iptables-alerter-jtt6k\" (UID: \"acc4af26-c5e0-47f8-bfaf-28d1927fd13d\") " pod="openshift-network-operator/iptables-alerter-jtt6k" Sep 11 18:48:17.076532 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.076469 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgvsw\" (UniqueName: \"kubernetes.io/projected/16a827b3-ee19-4675-ab27-8b47231c2ebc-kube-api-access-jgvsw\") pod \"node-resolver-n8gcj\" (UID: \"16a827b3-ee19-4675-ab27-8b47231c2ebc\") " pod="openshift-dns/node-resolver-n8gcj" Sep 11 18:48:17.079030 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.079010 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6kfj\" (UniqueName: \"kubernetes.io/projected/112256f4-7563-4c95-8204-1f2fee8e0b8a-kube-api-access-j6kfj\") pod \"aws-ebs-csi-driver-node-thv7n\" (UID: \"112256f4-7563-4c95-8204-1f2fee8e0b8a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:17.079362 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.079341 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9lc7\" (UniqueName: \"kubernetes.io/projected/977bd35b-33a1-4762-bfb1-7645b9dae842-kube-api-access-r9lc7\") pod \"tuned-46g48\" (UID: \"977bd35b-33a1-4762-bfb1-7645b9dae842\") " pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.168855 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.168823 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fb6ab28-29cd-45d4-8057-68bc5875c32e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.169034 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.168863 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8qlk\" (UniqueName: \"kubernetes.io/projected/2fb6ab28-29cd-45d4-8057-68bc5875c32e-kube-api-access-x8qlk\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.169034 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.168888 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-var-lib-cni-multus\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.169034 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.168911 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-kubelet\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.169034 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.168931 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-node-log\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.169034 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.168961 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-hostroot\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.169034 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.168983 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-log-socket\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.169034 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169003 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad502b23-3af5-4907-8f22-1c3c619555ca-konnectivity-ca\") pod \"konnectivity-agent-4gnvd\" (UID: \"ad502b23-3af5-4907-8f22-1c3c619555ca\") " pod="kube-system/konnectivity-agent-4gnvd" Sep 11 18:48:17.169034 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169001 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-var-lib-cni-multus\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.169034 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169027 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-run-netns\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169049 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-systemd-units\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169070 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70e064d0-8173-4c6e-b32f-a903dbad84cc-ovnkube-script-lib\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169098 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169126 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-cnibin\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169151 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-etc-openvswitch\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169176 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fb6ab28-29cd-45d4-8057-68bc5875c32e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169202 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/864dee94-a189-44d1-a4ec-00fa82d5d67e-cni-binary-copy\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169233 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-run-systemd\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169257 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-cni-bin\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169277 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-systemd-units\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169299 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-hostroot\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169283 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169331 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-log-socket\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169359 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fb6ab28-29cd-45d4-8057-68bc5875c32e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169388 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fb6ab28-29cd-45d4-8057-68bc5875c32e-cnibin\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169431 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/908b068d-8d94-4fcb-8490-eb8b1ac59a90-host\") pod \"node-ca-vm7pc\" (UID: \"908b068d-8d94-4fcb-8490-eb8b1ac59a90\") " pod="openshift-image-registry/node-ca-vm7pc" Sep 11 18:48:17.169484 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169452 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-etc-kubernetes\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169452 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169474 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-cni-netd\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169497 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70e064d0-8173-4c6e-b32f-a903dbad84cc-env-overrides\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169522 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70e064d0-8173-4c6e-b32f-a903dbad84cc-ovn-node-metrics-cert\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169567 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-system-cni-dir\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169579 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-kubelet\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169594 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-var-lib-cni-bin\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169619 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-run-netns\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169641 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-cni-netd\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169663 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-system-cni-dir\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169665 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-run-ovn\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169705 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-run-ovn\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169709 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/908b068d-8d94-4fcb-8490-eb8b1ac59a90-host\") pod \"node-ca-vm7pc\" (UID: \"908b068d-8d94-4fcb-8490-eb8b1ac59a90\") " pod="openshift-image-registry/node-ca-vm7pc" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169722 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khnc2\" (UniqueName: \"kubernetes.io/projected/908b068d-8d94-4fcb-8490-eb8b1ac59a90-kube-api-access-khnc2\") pod \"node-ca-vm7pc\" (UID: \"908b068d-8d94-4fcb-8490-eb8b1ac59a90\") " pod="openshift-image-registry/node-ca-vm7pc" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169745 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fb6ab28-29cd-45d4-8057-68bc5875c32e-cnibin\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169748 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-multus-cni-dir\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169566 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fb6ab28-29cd-45d4-8057-68bc5875c32e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.170331 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169774 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-multus-socket-dir-parent\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169814 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-run-netns\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169861 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-multus-socket-dir-parent\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169887 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fb6ab28-29cd-45d4-8057-68bc5875c32e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169913 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-multus-cni-dir\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169939 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad502b23-3af5-4907-8f22-1c3c619555ca-konnectivity-ca\") pod \"konnectivity-agent-4gnvd\" (UID: \"ad502b23-3af5-4907-8f22-1c3c619555ca\") " pod="kube-system/konnectivity-agent-4gnvd" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169952 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169963 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70e064d0-8173-4c6e-b32f-a903dbad84cc-ovnkube-script-lib\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170005 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-etc-openvswitch\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170011 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-cnibin\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.169751 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-etc-kubernetes\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170029 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-run-netns\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170050 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-cni-bin\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170057 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-var-lib-cni-bin\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170092 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-run-systemd\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170095 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-node-log\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170090 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fb6ab28-29cd-45d4-8057-68bc5875c32e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170125 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-run-k8s-cni-cncf-io\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.171209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170146 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-var-lib-kubelet\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170162 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70e064d0-8173-4c6e-b32f-a903dbad84cc-env-overrides\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170173 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/908b068d-8d94-4fcb-8490-eb8b1ac59a90-serviceca\") pod \"node-ca-vm7pc\" (UID: \"908b068d-8d94-4fcb-8490-eb8b1ac59a90\") " pod="openshift-image-registry/node-ca-vm7pc" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170209 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/864dee94-a189-44d1-a4ec-00fa82d5d67e-multus-daemon-config\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170217 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-var-lib-kubelet\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170219 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-run-k8s-cni-cncf-io\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170236 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-multus-conf-dir\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170264 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-run-multus-certs\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170270 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/864dee94-a189-44d1-a4ec-00fa82d5d67e-cni-binary-copy\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170288 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkdtt\" (UniqueName: \"kubernetes.io/projected/864dee94-a189-44d1-a4ec-00fa82d5d67e-kube-api-access-lkdtt\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170312 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-run-openvswitch\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170321 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-multus-conf-dir\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170337 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fb6ab28-29cd-45d4-8057-68bc5875c32e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170343 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/864dee94-a189-44d1-a4ec-00fa82d5d67e-host-run-multus-certs\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170362 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-run-openvswitch\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170378 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70e064d0-8173-4c6e-b32f-a903dbad84cc-ovnkube-config\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170416 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfhjl\" (UniqueName: \"kubernetes.io/projected/70e064d0-8173-4c6e-b32f-a903dbad84cc-kube-api-access-qfhjl\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170440 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad502b23-3af5-4907-8f22-1c3c619555ca-agent-certs\") pod \"konnectivity-agent-4gnvd\" (UID: \"ad502b23-3af5-4907-8f22-1c3c619555ca\") " pod="kube-system/konnectivity-agent-4gnvd" Sep 11 18:48:17.172080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170462 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-slash\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.172861 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170483 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-var-lib-openvswitch\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.172861 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170509 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fb6ab28-29cd-45d4-8057-68bc5875c32e-system-cni-dir\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.172861 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170583 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fb6ab28-29cd-45d4-8057-68bc5875c32e-system-cni-dir\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.172861 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170605 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fb6ab28-29cd-45d4-8057-68bc5875c32e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.172861 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170617 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/908b068d-8d94-4fcb-8490-eb8b1ac59a90-serviceca\") pod \"node-ca-vm7pc\" (UID: \"908b068d-8d94-4fcb-8490-eb8b1ac59a90\") " pod="openshift-image-registry/node-ca-vm7pc" Sep 11 18:48:17.172861 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170624 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-host-slash\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.172861 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170713 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/864dee94-a189-44d1-a4ec-00fa82d5d67e-multus-daemon-config\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.172861 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170767 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70e064d0-8173-4c6e-b32f-a903dbad84cc-var-lib-openvswitch\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.172861 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.170824 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70e064d0-8173-4c6e-b32f-a903dbad84cc-ovnkube-config\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.172861 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.171177 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-253.ec2.internal" event={"ID":"bdd962729f012a9148b1a889b407d7c0","Type":"ContainerStarted","Data":"66c6bd739bcc8cb56c57171f3d92b278bd56cc1f5bd8f805459dac64373b996d"} Sep 11 18:48:17.173307 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.173051 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal" event={"ID":"98d66ef951df454b941123496bfb620c","Type":"ContainerStarted","Data":"05923aec19ecad20ea28073db5f0a96ebaadd9a500ef54d6a7f6ab895a941c73"} Sep 11 18:48:17.173453 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.173434 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70e064d0-8173-4c6e-b32f-a903dbad84cc-ovn-node-metrics-cert\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.174027 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.174005 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad502b23-3af5-4907-8f22-1c3c619555ca-agent-certs\") pod \"konnectivity-agent-4gnvd\" (UID: \"ad502b23-3af5-4907-8f22-1c3c619555ca\") " pod="kube-system/konnectivity-agent-4gnvd" Sep 11 18:48:17.180789 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.180757 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khnc2\" (UniqueName: \"kubernetes.io/projected/908b068d-8d94-4fcb-8490-eb8b1ac59a90-kube-api-access-khnc2\") pod \"node-ca-vm7pc\" (UID: \"908b068d-8d94-4fcb-8490-eb8b1ac59a90\") " pod="openshift-image-registry/node-ca-vm7pc" Sep 11 18:48:17.182075 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.182051 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8qlk\" (UniqueName: \"kubernetes.io/projected/2fb6ab28-29cd-45d4-8057-68bc5875c32e-kube-api-access-x8qlk\") pod \"multus-additional-cni-plugins-lphqn\" (UID: \"2fb6ab28-29cd-45d4-8057-68bc5875c32e\") " pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.182903 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.182877 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkdtt\" (UniqueName: \"kubernetes.io/projected/864dee94-a189-44d1-a4ec-00fa82d5d67e-kube-api-access-lkdtt\") pod \"multus-thztx\" (UID: \"864dee94-a189-44d1-a4ec-00fa82d5d67e\") " pod="openshift-multus/multus-thztx" Sep 11 18:48:17.183138 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.183096 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfhjl\" (UniqueName: \"kubernetes.io/projected/70e064d0-8173-4c6e-b32f-a903dbad84cc-kube-api-access-qfhjl\") pod \"ovnkube-node-42nvj\" (UID: \"70e064d0-8173-4c6e-b32f-a903dbad84cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.258015 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.257984 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-46g48" Sep 11 18:48:17.269963 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.269940 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n8gcj" Sep 11 18:48:17.277721 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.277701 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" Sep 11 18:48:17.285422 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.285404 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jtt6k" Sep 11 18:48:17.291866 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.291850 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vm7pc" Sep 11 18:48:17.301461 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.301441 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-thztx" Sep 11 18:48:17.307997 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.307981 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:17.315645 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.315623 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4gnvd" Sep 11 18:48:17.321210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.321188 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lphqn" Sep 11 18:48:17.573247 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.573163 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret\") pod \"global-pull-secret-syncer-r2xrg\" (UID: \"953eaca9-ab99-4bce-b95a-ab3d3fbad183\") " pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:17.573247 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.573223 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs\") pod \"network-metrics-daemon-rhmpv\" (UID: \"681500e4-89ea-42a9-9d91-0d44e7698747\") " pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:17.573469 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:17.573336 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:48:17.573469 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:17.573348 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:48:17.573469 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:17.573415 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret podName:953eaca9-ab99-4bce-b95a-ab3d3fbad183 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:18.573396689 +0000 UTC m=+4.027202638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret") pod "global-pull-secret-syncer-r2xrg" (UID: "953eaca9-ab99-4bce-b95a-ab3d3fbad183") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:48:17.573469 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:17.573436 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs podName:681500e4-89ea-42a9-9d91-0d44e7698747 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:18.573425535 +0000 UTC m=+4.027231468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs") pod "network-metrics-daemon-rhmpv" (UID: "681500e4-89ea-42a9-9d91-0d44e7698747") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:48:17.673675 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:17.673639 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtg7v\" (UniqueName: \"kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v\") pod \"network-check-target-grcgm\" (UID: \"fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0\") " pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:17.673872 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:17.673804 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:48:17.673872 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:17.673822 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:48:17.673872 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:17.673830 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mtg7v for pod openshift-network-diagnostics/network-check-target-grcgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:48:17.674029 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:17.673885 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v podName:fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:18.673871717 +0000 UTC m=+4.127677654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtg7v" (UniqueName: "kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v") pod "network-check-target-grcgm" (UID: "fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:48:18.039600 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:18.039498 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:43:17 +0000 UTC" deadline="2028-04-25 21:48:33.442369215 +0000 UTC" Sep 11 18:48:18.039600 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:18.039533 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14211h0m15.402840773s" Sep 11 18:48:18.165393 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:18.165323 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:18.165570 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:18.165472 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:18.165821 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:18.165347 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:18.165821 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:18.165777 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:18.208112 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:18.207971 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad502b23_3af5_4907_8f22_1c3c619555ca.slice/crio-d3e77856e72b525ac0cc8de55a50f30bcde180d1cae30fdeea6d0ec62c3c6764 WatchSource:0}: Error finding container d3e77856e72b525ac0cc8de55a50f30bcde180d1cae30fdeea6d0ec62c3c6764: Status 404 returned error can't find the container with id d3e77856e72b525ac0cc8de55a50f30bcde180d1cae30fdeea6d0ec62c3c6764 Sep 11 18:48:18.209535 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:18.209512 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc4af26_c5e0_47f8_bfaf_28d1927fd13d.slice/crio-9670c0da420bd407af632f4894d5cce43fd63028fbd02d874a47f5ed8e725716 WatchSource:0}: Error finding container 9670c0da420bd407af632f4894d5cce43fd63028fbd02d874a47f5ed8e725716: Status 404 returned error can't find the container with id 9670c0da420bd407af632f4894d5cce43fd63028fbd02d874a47f5ed8e725716 Sep 11 18:48:18.213702 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:18.213678 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb6ab28_29cd_45d4_8057_68bc5875c32e.slice/crio-dbeb5592e800a51f00e89bd436a0208bdd5aa875c88dcde32ada3fc2301cfd3f WatchSource:0}: Error finding container dbeb5592e800a51f00e89bd436a0208bdd5aa875c88dcde32ada3fc2301cfd3f: Status 404 returned error can't find the container with id dbeb5592e800a51f00e89bd436a0208bdd5aa875c88dcde32ada3fc2301cfd3f Sep 11 18:48:18.214261 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:18.214244 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112256f4_7563_4c95_8204_1f2fee8e0b8a.slice/crio-05ffb0f3136a9b725d6573c91dfb4efdc1be9b30b989bddd0deb117c61232bca WatchSource:0}: Error finding container 05ffb0f3136a9b725d6573c91dfb4efdc1be9b30b989bddd0deb117c61232bca: Status 404 returned error can't find the container with id 05ffb0f3136a9b725d6573c91dfb4efdc1be9b30b989bddd0deb117c61232bca Sep 11 18:48:18.215896 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:18.215769 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e064d0_8173_4c6e_b32f_a903dbad84cc.slice/crio-1e5e791288abe19700ce3e1571723283c6a51441b3a658d712ea3995bc547e55 WatchSource:0}: Error finding container 1e5e791288abe19700ce3e1571723283c6a51441b3a658d712ea3995bc547e55: Status 404 returned error can't find the container with id 1e5e791288abe19700ce3e1571723283c6a51441b3a658d712ea3995bc547e55 Sep 11 18:48:18.216628 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:18.216607 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864dee94_a189_44d1_a4ec_00fa82d5d67e.slice/crio-bc624dcb92e8b81b1a333000de1854ab38084ab93f1a48ad3c2690866800e561 WatchSource:0}: Error finding container bc624dcb92e8b81b1a333000de1854ab38084ab93f1a48ad3c2690866800e561: Status 404 returned error can't find the container with id bc624dcb92e8b81b1a333000de1854ab38084ab93f1a48ad3c2690866800e561 Sep 11 18:48:18.217955 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:18.217462 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908b068d_8d94_4fcb_8490_eb8b1ac59a90.slice/crio-b971ec8f70005c82faa7544f8770b48a74dee1dbf94d642de7a0eccb41b3cb74 WatchSource:0}: Error finding container b971ec8f70005c82faa7544f8770b48a74dee1dbf94d642de7a0eccb41b3cb74: Status 404 returned error can't find the container with id b971ec8f70005c82faa7544f8770b48a74dee1dbf94d642de7a0eccb41b3cb74 Sep 11 18:48:18.219053 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:18.219027 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a827b3_ee19_4675_ab27_8b47231c2ebc.slice/crio-1ab7230ba820197afad069a5659fefe87d75cf376ac20019cc98ecdd6c28c478 WatchSource:0}: Error finding container 1ab7230ba820197afad069a5659fefe87d75cf376ac20019cc98ecdd6c28c478: Status 404 returned error can't find the container with id 1ab7230ba820197afad069a5659fefe87d75cf376ac20019cc98ecdd6c28c478 Sep 11 18:48:18.219790 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:18.219747 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977bd35b_33a1_4762_bfb1_7645b9dae842.slice/crio-9906f144fdbfa6717d1d87784d578b57082bec411c74bc18bec8c14f4a7062ac WatchSource:0}: Error finding container 9906f144fdbfa6717d1d87784d578b57082bec411c74bc18bec8c14f4a7062ac: Status 404 returned error can't find the container with id 9906f144fdbfa6717d1d87784d578b57082bec411c74bc18bec8c14f4a7062ac Sep 11 18:48:18.580325 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:18.580039 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret\") pod \"global-pull-secret-syncer-r2xrg\" (UID: \"953eaca9-ab99-4bce-b95a-ab3d3fbad183\") " pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:18.580465 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:18.580330 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs\") pod \"network-metrics-daemon-rhmpv\" (UID: \"681500e4-89ea-42a9-9d91-0d44e7698747\") " pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:18.580465 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:18.580211 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:48:18.580465 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:18.580437 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret podName:953eaca9-ab99-4bce-b95a-ab3d3fbad183 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:20.580416457 +0000 UTC m=+6.034222390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret") pod "global-pull-secret-syncer-r2xrg" (UID: "953eaca9-ab99-4bce-b95a-ab3d3fbad183") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:48:18.580465 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:18.580440 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:48:18.580687 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:18.580501 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs podName:681500e4-89ea-42a9-9d91-0d44e7698747 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:20.58048549 +0000 UTC m=+6.034291444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs") pod "network-metrics-daemon-rhmpv" (UID: "681500e4-89ea-42a9-9d91-0d44e7698747") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:48:18.681673 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:18.681640 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtg7v\" (UniqueName: \"kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v\") pod \"network-check-target-grcgm\" (UID: \"fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0\") " pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:18.681851 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:18.681796 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:48:18.681851 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:18.681810 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:48:18.681851 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:18.681820 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mtg7v for pod openshift-network-diagnostics/network-check-target-grcgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:48:18.682016 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:18.681863 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v podName:fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:20.681851733 +0000 UTC m=+6.135657666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtg7v" (UniqueName: "kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v") pod "network-check-target-grcgm" (UID: "fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:48:19.040525 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:19.040402 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:43:17 +0000 UTC" deadline="2028-06-22 09:59:39.062414922 +0000 UTC" Sep 11 18:48:19.040525 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:19.040466 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15591h11m20.021953463s" Sep 11 18:48:19.168993 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:19.168479 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:19.168993 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:19.168618 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:19.182172 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:19.182141 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lphqn" event={"ID":"2fb6ab28-29cd-45d4-8057-68bc5875c32e","Type":"ContainerStarted","Data":"dbeb5592e800a51f00e89bd436a0208bdd5aa875c88dcde32ada3fc2301cfd3f"} Sep 11 18:48:19.196284 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:19.196176 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4gnvd" event={"ID":"ad502b23-3af5-4907-8f22-1c3c619555ca","Type":"ContainerStarted","Data":"d3e77856e72b525ac0cc8de55a50f30bcde180d1cae30fdeea6d0ec62c3c6764"} Sep 11 18:48:19.208727 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:19.208083 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-253.ec2.internal" event={"ID":"bdd962729f012a9148b1a889b407d7c0","Type":"ContainerStarted","Data":"f9e42f83a6806d32b4a0c34dbcdc0a399481fa437fdb66ef7e67ad789a5130c2"} Sep 11 18:48:19.226062 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:19.225514 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-253.ec2.internal" podStartSLOduration=3.225496027 podStartE2EDuration="3.225496027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:19.224951527 +0000 UTC m=+4.678757484" watchObservedRunningTime="2026-09-11 18:48:19.225496027 +0000 UTC m=+4.679301987" Sep 11 18:48:19.226062 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:19.225581 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal" event={"ID":"98d66ef951df454b941123496bfb620c","Type":"ContainerDied","Data":"03d44de400a802f7fe5e06d3869473d4c4e5304bbdf0970b69483de6c25903b8"} Sep 11 18:48:19.226062 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:19.225532 2556 generic.go:356] "Generic (PLEG): container finished" podID="98d66ef951df454b941123496bfb620c" containerID="03d44de400a802f7fe5e06d3869473d4c4e5304bbdf0970b69483de6c25903b8" exitCode=0 Sep 11 18:48:19.239196 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:19.239115 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vm7pc" event={"ID":"908b068d-8d94-4fcb-8490-eb8b1ac59a90","Type":"ContainerStarted","Data":"b971ec8f70005c82faa7544f8770b48a74dee1dbf94d642de7a0eccb41b3cb74"} Sep 11 18:48:19.242814 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:19.242690 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-thztx" event={"ID":"864dee94-a189-44d1-a4ec-00fa82d5d67e","Type":"ContainerStarted","Data":"bc624dcb92e8b81b1a333000de1854ab38084ab93f1a48ad3c2690866800e561"} Sep 11 18:48:19.253032 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:19.252929 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" event={"ID":"70e064d0-8173-4c6e-b32f-a903dbad84cc","Type":"ContainerStarted","Data":"1e5e791288abe19700ce3e1571723283c6a51441b3a658d712ea3995bc547e55"} Sep 11 18:48:19.260077 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:19.260022 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jtt6k" event={"ID":"acc4af26-c5e0-47f8-bfaf-28d1927fd13d","Type":"ContainerStarted","Data":"9670c0da420bd407af632f4894d5cce43fd63028fbd02d874a47f5ed8e725716"} Sep 11 18:48:19.270361 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:19.270271 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-46g48" event={"ID":"977bd35b-33a1-4762-bfb1-7645b9dae842","Type":"ContainerStarted","Data":"9906f144fdbfa6717d1d87784d578b57082bec411c74bc18bec8c14f4a7062ac"} Sep 11 18:48:19.282921 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:19.282820 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n8gcj" event={"ID":"16a827b3-ee19-4675-ab27-8b47231c2ebc","Type":"ContainerStarted","Data":"1ab7230ba820197afad069a5659fefe87d75cf376ac20019cc98ecdd6c28c478"} Sep 11 18:48:19.292341 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:19.292276 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" event={"ID":"112256f4-7563-4c95-8204-1f2fee8e0b8a","Type":"ContainerStarted","Data":"05ffb0f3136a9b725d6573c91dfb4efdc1be9b30b989bddd0deb117c61232bca"} Sep 11 18:48:20.165514 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:20.165431 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:20.166220 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:20.165609 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:20.166220 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:20.166051 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:20.166220 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:20.166180 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:20.321663 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:20.321631 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal" event={"ID":"98d66ef951df454b941123496bfb620c","Type":"ContainerStarted","Data":"a5942e75719bfdf4cabcd1be0ebebe147d45d9f747f53f035bd5c2898f9b1484"} Sep 11 18:48:20.335953 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:20.335895 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-253.ec2.internal" podStartSLOduration=4.3358736669999995 podStartE2EDuration="4.335873667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:20.335534068 +0000 UTC m=+5.789340024" watchObservedRunningTime="2026-09-11 18:48:20.335873667 +0000 UTC m=+5.789679624" Sep 11 18:48:20.600145 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:20.599886 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret\") pod \"global-pull-secret-syncer-r2xrg\" (UID: \"953eaca9-ab99-4bce-b95a-ab3d3fbad183\") " pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:20.600145 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:20.599946 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs\") pod \"network-metrics-daemon-rhmpv\" (UID: \"681500e4-89ea-42a9-9d91-0d44e7698747\") " pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:20.600145 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:20.600071 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:48:20.600145 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:20.600136 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs podName:681500e4-89ea-42a9-9d91-0d44e7698747 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:24.600113981 +0000 UTC m=+10.053919933 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs") pod "network-metrics-daemon-rhmpv" (UID: "681500e4-89ea-42a9-9d91-0d44e7698747") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:48:20.600634 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:20.600608 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:48:20.600704 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:20.600665 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret podName:953eaca9-ab99-4bce-b95a-ab3d3fbad183 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:24.600649437 +0000 UTC m=+10.054455384 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret") pod "global-pull-secret-syncer-r2xrg" (UID: "953eaca9-ab99-4bce-b95a-ab3d3fbad183") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:48:20.701063 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:20.701023 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtg7v\" (UniqueName: \"kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v\") pod \"network-check-target-grcgm\" (UID: \"fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0\") " pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:20.701265 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:20.701247 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:48:20.701354 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:20.701272 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:48:20.701354 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:20.701286 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mtg7v for pod openshift-network-diagnostics/network-check-target-grcgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:48:20.701354 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:20.701346 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v podName:fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:24.701328618 +0000 UTC m=+10.155134567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtg7v" (UniqueName: "kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v") pod "network-check-target-grcgm" (UID: "fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:48:21.168344 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:21.168313 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:21.168834 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:21.168448 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:22.166189 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:22.166154 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:22.166372 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:22.166281 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:22.166821 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:22.166667 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:22.166821 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:22.166775 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:23.166477 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:23.166001 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:23.166477 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:23.166139 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:24.165921 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:24.165880 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:24.166079 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:24.166029 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:24.166420 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:24.166403 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:24.166520 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:24.166509 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:24.632812 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:24.632716 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret\") pod \"global-pull-secret-syncer-r2xrg\" (UID: \"953eaca9-ab99-4bce-b95a-ab3d3fbad183\") " pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:24.632812 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:24.632777 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs\") pod \"network-metrics-daemon-rhmpv\" (UID: \"681500e4-89ea-42a9-9d91-0d44e7698747\") " pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:24.633026 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:24.632889 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:48:24.633026 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:24.632896 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:48:24.633026 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:24.632946 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs podName:681500e4-89ea-42a9-9d91-0d44e7698747 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:32.632927649 +0000 UTC m=+18.086733582 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs") pod "network-metrics-daemon-rhmpv" (UID: "681500e4-89ea-42a9-9d91-0d44e7698747") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:48:24.633026 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:24.632964 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret podName:953eaca9-ab99-4bce-b95a-ab3d3fbad183 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:32.632956627 +0000 UTC m=+18.086762560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret") pod "global-pull-secret-syncer-r2xrg" (UID: "953eaca9-ab99-4bce-b95a-ab3d3fbad183") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:48:24.733926 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:24.733878 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtg7v\" (UniqueName: \"kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v\") pod \"network-check-target-grcgm\" (UID: \"fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0\") " pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:24.734112 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:24.734041 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:48:24.734112 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:24.734062 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:48:24.734112 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:24.734073 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mtg7v for pod openshift-network-diagnostics/network-check-target-grcgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:48:24.734247 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:24.734139 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v podName:fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:32.734119992 +0000 UTC m=+18.187925927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtg7v" (UniqueName: "kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v") pod "network-check-target-grcgm" (UID: "fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:48:25.166405 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:25.166372 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:25.166574 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:25.166486 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:26.165327 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:26.165280 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:26.165510 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:26.165414 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:26.165783 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:26.165280 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:26.165783 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:26.165747 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:27.165923 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:27.165714 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:27.165923 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:27.165835 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:28.166093 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:28.166050 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:28.166535 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:28.166050 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:28.166535 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:28.166209 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:28.166535 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:28.166297 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:29.165874 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:29.165836 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:29.166055 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:29.165955 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:30.165383 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:30.165351 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:30.165874 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:30.165356 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:30.165874 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:30.165481 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:30.165874 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:30.165590 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:31.165839 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:31.165804 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:31.166268 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:31.165922 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:32.165709 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:32.165663 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:32.165940 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:32.165809 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:32.165940 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:32.165867 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:32.166321 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:32.165975 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:32.693601 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:32.693566 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret\") pod \"global-pull-secret-syncer-r2xrg\" (UID: \"953eaca9-ab99-4bce-b95a-ab3d3fbad183\") " pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:32.693787 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:32.693615 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs\") pod \"network-metrics-daemon-rhmpv\" (UID: \"681500e4-89ea-42a9-9d91-0d44e7698747\") " pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:32.693787 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:32.693729 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:48:32.693787 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:32.693734 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:48:32.693787 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:32.693785 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs podName:681500e4-89ea-42a9-9d91-0d44e7698747 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:48.69377028 +0000 UTC m=+34.147576213 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs") pod "network-metrics-daemon-rhmpv" (UID: "681500e4-89ea-42a9-9d91-0d44e7698747") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:48:32.693983 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:32.693803 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret podName:953eaca9-ab99-4bce-b95a-ab3d3fbad183 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:48.693793969 +0000 UTC m=+34.147599906 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret") pod "global-pull-secret-syncer-r2xrg" (UID: "953eaca9-ab99-4bce-b95a-ab3d3fbad183") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:48:32.794973 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:32.794936 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtg7v\" (UniqueName: \"kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v\") pod \"network-check-target-grcgm\" (UID: \"fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0\") " pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:32.795156 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:32.795128 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:48:32.795156 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:32.795150 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:48:32.795267 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:32.795159 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mtg7v for pod openshift-network-diagnostics/network-check-target-grcgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:48:32.795267 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:32.795210 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v podName:fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:48.795196093 +0000 UTC m=+34.249002029 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtg7v" (UniqueName: "kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v") pod "network-check-target-grcgm" (UID: "fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:48:33.165807 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:33.165517 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:33.165980 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:33.165867 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:34.165499 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:34.165458 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:34.165691 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:34.165477 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:34.165691 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:34.165607 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:34.165691 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:34.165672 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:35.166239 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:35.166206 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:35.166685 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:35.166311 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:36.165614 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.165583 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:36.165614 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.165614 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:36.165794 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:36.165689 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:36.165867 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:36.165833 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:36.358575 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.358524 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" event={"ID":"112256f4-7563-4c95-8204-1f2fee8e0b8a","Type":"ContainerStarted","Data":"79019b5baba5ebc7dea65ffe7080b364f573649baa2d3037b88a623ef6023e15"} Sep 11 18:48:36.360486 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.360432 2556 generic.go:356] "Generic (PLEG): container finished" podID="2fb6ab28-29cd-45d4-8057-68bc5875c32e" containerID="d6ef7ef966bd193d19a66aa764fd4f2f4b74c3092312d4340e65da112f85b38f" exitCode=0 Sep 11 18:48:36.360620 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.360533 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lphqn" event={"ID":"2fb6ab28-29cd-45d4-8057-68bc5875c32e","Type":"ContainerDied","Data":"d6ef7ef966bd193d19a66aa764fd4f2f4b74c3092312d4340e65da112f85b38f"} Sep 11 18:48:36.362524 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.362494 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4gnvd" event={"ID":"ad502b23-3af5-4907-8f22-1c3c619555ca","Type":"ContainerStarted","Data":"c43ef855699d605607db79861f766df9b9b439311e232eef7400b1202c5a23ee"} Sep 11 18:48:36.364450 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.364421 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vm7pc" event={"ID":"908b068d-8d94-4fcb-8490-eb8b1ac59a90","Type":"ContainerStarted","Data":"62b58a4097f10b2b0c399b7ccc1252c382a4c8cf7777d9cb780fe4f535ca7c7d"} Sep 11 18:48:36.366388 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.366326 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-thztx" event={"ID":"864dee94-a189-44d1-a4ec-00fa82d5d67e","Type":"ContainerStarted","Data":"1c8ef7a51cb0c35f23b2ae10788dba42ad0f698b1491ca91200dc0b05eac378a"} Sep 11 18:48:36.368351 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.368329 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 18:48:36.368763 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.368697 2556 generic.go:356] "Generic (PLEG): container finished" podID="70e064d0-8173-4c6e-b32f-a903dbad84cc" containerID="9ea0d0225909e2e46f3a15deb37280685c56b7b81ab933ec6f453cddd3b124c4" exitCode=1 Sep 11 18:48:36.368848 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.368799 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" event={"ID":"70e064d0-8173-4c6e-b32f-a903dbad84cc","Type":"ContainerDied","Data":"9ea0d0225909e2e46f3a15deb37280685c56b7b81ab933ec6f453cddd3b124c4"} Sep 11 18:48:36.368904 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.368870 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" event={"ID":"70e064d0-8173-4c6e-b32f-a903dbad84cc","Type":"ContainerStarted","Data":"89fc347f4c1edc4c153e71a53cf0c759b6a018e5762f32747ec0b5cea418eaf9"} Sep 11 18:48:36.370643 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.370617 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-46g48" event={"ID":"977bd35b-33a1-4762-bfb1-7645b9dae842","Type":"ContainerStarted","Data":"49ea24797bf0d80f4935ef4256cba0b0eda7aacf0f1f9b5b2130c90ef4b6f539"} Sep 11 18:48:36.373926 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.373900 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n8gcj" event={"ID":"16a827b3-ee19-4675-ab27-8b47231c2ebc","Type":"ContainerStarted","Data":"70119491cdb9f971269260dcc72600ed8bd93764cfe1b20225a54f26dfdecb17"} Sep 11 18:48:36.407730 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.407562 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-46g48" podStartSLOduration=3.9052612780000002 podStartE2EDuration="21.407518431s" totalImagesPullingTime="17.502257153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:36.406971182 +0000 UTC m=+21.860777138" watchObservedRunningTime="2026-09-11 18:48:36.407518431 +0000 UTC m=+21.861324387" Sep 11 18:48:36.421788 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.421740 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vm7pc" podStartSLOduration=3.948415167 podStartE2EDuration="21.421727251s" totalImagesPullingTime="17.473312084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:36.421646021 +0000 UTC m=+21.875451998" watchObservedRunningTime="2026-09-11 18:48:36.421727251 +0000 UTC m=+21.875533205" Sep 11 18:48:36.441721 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.441656 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-thztx" podStartSLOduration=3.899767394 podStartE2EDuration="21.441638386s" totalImagesPullingTime="17.541870992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:36.440859127 +0000 UTC m=+21.894665100" watchObservedRunningTime="2026-09-11 18:48:36.441638386 +0000 UTC m=+21.895444333" Sep 11 18:48:36.490182 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.490104 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4gnvd" podStartSLOduration=4.260530591 podStartE2EDuration="21.490086721s" totalImagesPullingTime="17.22955613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:36.473513906 +0000 UTC m=+21.927319879" watchObservedRunningTime="2026-09-11 18:48:36.490086721 +0000 UTC m=+21.943892699" Sep 11 18:48:36.490323 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:36.490209 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n8gcj" podStartSLOduration=3.986257957 podStartE2EDuration="21.490202106s" totalImagesPullingTime="17.503944149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:36.489956849 +0000 UTC m=+21.943762823" watchObservedRunningTime="2026-09-11 18:48:36.490202106 +0000 UTC m=+21.944008086" Sep 11 18:48:37.078990 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:37.078954 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4gnvd" Sep 11 18:48:37.079621 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:37.079602 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4gnvd" Sep 11 18:48:37.165321 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:37.165300 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:37.165446 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:37.165412 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:37.177433 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:37.177407 2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 18:48:37.378756 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:37.378693 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 18:48:37.379372 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:37.379100 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" event={"ID":"70e064d0-8173-4c6e-b32f-a903dbad84cc","Type":"ContainerStarted","Data":"6bee8965135210be1c770fcef68d3f11ff11f24c607ff53e2eab83afcfa8efdd"} Sep 11 18:48:37.379372 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:37.379205 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" event={"ID":"70e064d0-8173-4c6e-b32f-a903dbad84cc","Type":"ContainerStarted","Data":"130b1beb471f81f7b66279e96239ff3c88b139d9152bf48d91c69a17f0a71d4e"} Sep 11 18:48:37.379372 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:37.379243 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" event={"ID":"70e064d0-8173-4c6e-b32f-a903dbad84cc","Type":"ContainerStarted","Data":"1142074f52c33a43e5ff54358450ac29cacc2dcdc06835e11218ad1858db3930"} Sep 11 18:48:37.379372 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:37.379280 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" event={"ID":"70e064d0-8173-4c6e-b32f-a903dbad84cc","Type":"ContainerStarted","Data":"05098fa5ab57ff5f859f6d20f90bf0f3fc75eaec3b932ed134f9a807fa66bc6c"} Sep 11 18:48:37.380463 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:37.380441 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jtt6k" event={"ID":"acc4af26-c5e0-47f8-bfaf-28d1927fd13d","Type":"ContainerStarted","Data":"b10b9abcc7c39bab0e05181e79c824d948c1d874cdd4b36d63b497c6250a1239"} Sep 11 18:48:37.382232 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:37.382200 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" event={"ID":"112256f4-7563-4c95-8204-1f2fee8e0b8a","Type":"ContainerStarted","Data":"c5b1fb37898ad77ff33ed7d0ffeeff8e320f09c13aa639893b7d6a3453cd3a14"} Sep 11 18:48:37.382784 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:37.382755 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4gnvd" Sep 11 18:48:37.383332 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:37.383317 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4gnvd" Sep 11 18:48:37.395099 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:37.395062 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jtt6k" podStartSLOduration=4.926908416 podStartE2EDuration="22.395050027s" totalImagesPullingTime="17.468141611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:37.394872087 +0000 UTC m=+22.848678064" watchObservedRunningTime="2026-09-11 18:48:37.395050027 +0000 UTC m=+22.848855982" Sep 11 18:48:38.019738 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:38.019588 2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T18:48:37.177432301Z","UUID":"73bf6c25-1a46-4c60-817e-7922beca57c9","Handler":null,"Name":"","Endpoint":""} Sep 11 18:48:38.021330 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:38.021308 2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 18:48:38.021460 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:38.021344 2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 18:48:38.166027 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:38.165999 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:38.166027 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:38.166027 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:38.166244 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:38.166110 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:38.166304 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:38.166252 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:38.386487 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:38.386452 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" event={"ID":"112256f4-7563-4c95-8204-1f2fee8e0b8a","Type":"ContainerStarted","Data":"507b8d7edcec9a22f7d5a4c80707f54b21dfc55245587a00649a6bc635aeb9e6"} Sep 11 18:48:38.406556 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:38.406502 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-thv7n" podStartSLOduration=3.764200832 podStartE2EDuration="23.40648732s" totalImagesPullingTime="19.642286488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:38.405720921 +0000 UTC m=+23.859526886" watchObservedRunningTime="2026-09-11 18:48:38.40648732 +0000 UTC m=+23.860293276" Sep 11 18:48:39.165417 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:39.165386 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:39.165595 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:39.165557 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:39.392026 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:39.391961 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 18:48:39.392465 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:39.392359 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" event={"ID":"70e064d0-8173-4c6e-b32f-a903dbad84cc","Type":"ContainerStarted","Data":"a2504ec02910c30f59d7db13920db2763648256533beff77c5008bacdcb49733"} Sep 11 18:48:40.165086 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:40.165055 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:40.165266 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:40.165071 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:40.165679 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:40.165646 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:40.168249 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:40.166248 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:41.165190 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:41.165162 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:41.165527 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:41.165274 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:41.401982 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:41.401568 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 18:48:41.401982 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:41.401936 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" event={"ID":"70e064d0-8173-4c6e-b32f-a903dbad84cc","Type":"ContainerStarted","Data":"50ae55c5f5f6399422c680d5bdf4de6332169977ff3dece9b89cd56747964724"} Sep 11 18:48:42.165654 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:42.165417 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:42.166045 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:42.165431 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:42.166045 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:42.165686 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:42.166045 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:42.165753 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:42.405032 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:42.404997 2556 generic.go:356] "Generic (PLEG): container finished" podID="2fb6ab28-29cd-45d4-8057-68bc5875c32e" containerID="573a8b7e6329864d84c2c53c56753792b44b1443f605b7000452e55045665a2d" exitCode=0 Sep 11 18:48:42.405209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:42.405081 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lphqn" event={"ID":"2fb6ab28-29cd-45d4-8057-68bc5875c32e","Type":"ContainerDied","Data":"573a8b7e6329864d84c2c53c56753792b44b1443f605b7000452e55045665a2d"} Sep 11 18:48:42.405396 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:42.405382 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:42.405504 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:42.405490 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:42.405574 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:42.405519 2556 scope.go:117] "RemoveContainer" containerID="9ea0d0225909e2e46f3a15deb37280685c56b7b81ab933ec6f453cddd3b124c4" Sep 11 18:48:42.405574 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:42.405566 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:42.421079 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:42.421055 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:42.421404 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:42.421384 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:48:43.168695 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:43.168671 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:43.169161 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:43.168762 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:43.363917 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:43.363884 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rhmpv"] Sep 11 18:48:43.364088 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:43.364041 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:43.364184 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:43.364165 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:43.367926 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:43.367899 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r2xrg"] Sep 11 18:48:43.368056 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:43.367993 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:43.368122 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:43.368084 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:43.373328 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:43.373310 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-grcgm"] Sep 11 18:48:43.415201 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:43.415178 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 18:48:43.415579 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:43.415552 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" event={"ID":"70e064d0-8173-4c6e-b32f-a903dbad84cc","Type":"ContainerStarted","Data":"9ffea1a41e8b0d85912557b83c224a96eeeb2c0b342a6e856730d93b641143fc"} Sep 11 18:48:43.415663 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:43.415565 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:43.415927 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:43.415900 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:43.444187 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:43.444131 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" podStartSLOduration=10.621280521 podStartE2EDuration="28.444112163s" totalImagesPullingTime="17.822831642s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:43.442868734 +0000 UTC m=+28.896674691" watchObservedRunningTime="2026-09-11 18:48:43.444112163 +0000 UTC m=+28.897918119" Sep 11 18:48:44.420344 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:44.420308 2556 generic.go:356] "Generic (PLEG): container finished" podID="2fb6ab28-29cd-45d4-8057-68bc5875c32e" containerID="431d3233440d06b3d88d3597c4374dc20694d1b6de97b6cd5fa91230f4e80afa" exitCode=0 Sep 11 18:48:44.421080 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:44.420385 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lphqn" event={"ID":"2fb6ab28-29cd-45d4-8057-68bc5875c32e","Type":"ContainerDied","Data":"431d3233440d06b3d88d3597c4374dc20694d1b6de97b6cd5fa91230f4e80afa"} Sep 11 18:48:45.167476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:45.166937 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:45.167476 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:45.167059 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:45.167476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:45.167476 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:45.167767 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:45.167588 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:45.167767 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:45.167645 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:45.167767 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:45.167715 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:45.425246 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:45.425171 2556 generic.go:356] "Generic (PLEG): container finished" podID="2fb6ab28-29cd-45d4-8057-68bc5875c32e" containerID="796e777f1225c855974333a5c5b05464cdaf9a8a6af1922f24d2832a452c32b4" exitCode=0 Sep 11 18:48:45.425246 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:45.425220 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lphqn" event={"ID":"2fb6ab28-29cd-45d4-8057-68bc5875c32e","Type":"ContainerDied","Data":"796e777f1225c855974333a5c5b05464cdaf9a8a6af1922f24d2832a452c32b4"} Sep 11 18:48:47.165986 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:47.165782 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:47.166497 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:47.165782 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:47.166497 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:47.166103 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:47.166497 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:47.165797 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:47.166497 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:47.166154 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:47.166497 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:47.166224 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:48.707439 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:48.707407 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret\") pod \"global-pull-secret-syncer-r2xrg\" (UID: \"953eaca9-ab99-4bce-b95a-ab3d3fbad183\") " pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:48.707439 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:48.707446 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs\") pod \"network-metrics-daemon-rhmpv\" (UID: \"681500e4-89ea-42a9-9d91-0d44e7698747\") " pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:48.707852 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:48.707576 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:48:48.707852 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:48.707582 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 18:48:48.707852 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:48.707625 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs podName:681500e4-89ea-42a9-9d91-0d44e7698747 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:20.707611362 +0000 UTC m=+66.161417296 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs") pod "network-metrics-daemon-rhmpv" (UID: "681500e4-89ea-42a9-9d91-0d44e7698747") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 18:48:48.707852 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:48.707638 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret podName:953eaca9-ab99-4bce-b95a-ab3d3fbad183 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:20.707631961 +0000 UTC m=+66.161437894 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret") pod "global-pull-secret-syncer-r2xrg" (UID: "953eaca9-ab99-4bce-b95a-ab3d3fbad183") : object "kube-system"/"original-pull-secret" not registered Sep 11 18:48:48.808318 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:48.808294 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtg7v\" (UniqueName: \"kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v\") pod \"network-check-target-grcgm\" (UID: \"fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0\") " pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:48.808518 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:48.808497 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 18:48:48.808587 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:48.808529 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 18:48:48.808587 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:48.808558 2556 projected.go:196] Error preparing data for projected volume kube-api-access-mtg7v for pod openshift-network-diagnostics/network-check-target-grcgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:48:48.808654 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:48.808620 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v podName:fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:20.808602093 +0000 UTC m=+66.262408031 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtg7v" (UniqueName: "kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v") pod "network-check-target-grcgm" (UID: "fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 18:48:49.165467 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.165425 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:49.165762 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.165590 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:49.165762 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:49.165602 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rhmpv" podUID="681500e4-89ea-42a9-9d91-0d44e7698747" Sep 11 18:48:49.165762 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.165629 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:49.165762 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:49.165739 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grcgm" podUID="fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0" Sep 11 18:48:49.165983 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:49.165826 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2xrg" podUID="953eaca9-ab99-4bce-b95a-ab3d3fbad183" Sep 11 18:48:49.816708 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.816678 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-253.ec2.internal" event="NodeReady" Sep 11 18:48:49.817178 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.816807 2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 18:48:49.865097 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.864659 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6468cc4b89-rwvgm"] Sep 11 18:48:49.895588 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.895560 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h928z"] Sep 11 18:48:49.895759 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.895632 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:49.898553 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.898246 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 18:48:49.898553 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.898258 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 18:48:49.898732 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.898649 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pqdcd\"" Sep 11 18:48:49.899060 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.899039 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 18:48:49.904085 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.903956 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 18:48:49.909816 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.909789 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6468cc4b89-rwvgm"] Sep 11 18:48:49.909926 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.909827 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-g7ldv"] Sep 11 18:48:49.909926 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.909920 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h928z" Sep 11 18:48:49.912064 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.912007 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 18:48:49.912315 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.912295 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vlxp2\"" Sep 11 18:48:49.912425 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.912295 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 18:48:49.929628 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.929602 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h928z"] Sep 11 18:48:49.929742 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.929652 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g7ldv"] Sep 11 18:48:49.929798 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.929748 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g7ldv" Sep 11 18:48:49.932220 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.932193 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 18:48:49.932672 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.932602 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 18:48:49.932768 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.932720 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 18:48:49.932835 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:49.932822 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-shf8m\"" Sep 11 18:48:50.015693 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.015657 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.015693 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.015695 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-bound-sa-token\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.015933 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.015728 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-image-registry-private-configuration\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.015933 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.015753 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-ca-trust-extracted\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.015933 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.015778 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-459b4\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-kube-api-access-459b4\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.015933 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.015842 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bebc8327-abaa-4039-96f8-90928ef2e27d-config-volume\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:48:50.015933 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.015884 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-certificates\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.015933 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.015913 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:48:50.016238 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.015946 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-trusted-ca\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.016238 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.015972 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6j6j\" (UniqueName: \"kubernetes.io/projected/bebc8327-abaa-4039-96f8-90928ef2e27d-kube-api-access-h6j6j\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:48:50.016238 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.016003 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzkmr\" (UniqueName: \"kubernetes.io/projected/ad0303b1-5952-4ffc-b1d1-d256b97ba1e6-kube-api-access-gzkmr\") pod \"ingress-canary-g7ldv\" (UID: \"ad0303b1-5952-4ffc-b1d1-d256b97ba1e6\") " pod="openshift-ingress-canary/ingress-canary-g7ldv" Sep 11 18:48:50.016238 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.016030 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-installation-pull-secrets\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.016238 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.016055 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad0303b1-5952-4ffc-b1d1-d256b97ba1e6-cert\") pod \"ingress-canary-g7ldv\" (UID: \"ad0303b1-5952-4ffc-b1d1-d256b97ba1e6\") " pod="openshift-ingress-canary/ingress-canary-g7ldv" Sep 11 18:48:50.016238 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.016076 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bebc8327-abaa-4039-96f8-90928ef2e27d-tmp-dir\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:48:50.117520 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.117438 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.117520 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.117476 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-bound-sa-token\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.117520 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.117507 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-image-registry-private-configuration\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.117801 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.117532 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-ca-trust-extracted\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.117801 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.117568 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-459b4\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-kube-api-access-459b4\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.117801 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.117590 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bebc8327-abaa-4039-96f8-90928ef2e27d-config-volume\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:48:50.117801 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.117615 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-certificates\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.117801 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.117636 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:48:50.117801 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:50.117649 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 18:48:50.117801 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:50.117667 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6468cc4b89-rwvgm: secret "image-registry-tls" not found Sep 11 18:48:50.117801 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.117671 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-trusted-ca\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.117801 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.117693 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6j6j\" (UniqueName: \"kubernetes.io/projected/bebc8327-abaa-4039-96f8-90928ef2e27d-kube-api-access-h6j6j\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:48:50.117801 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:50.117728 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls podName:7c52ab3a-81f5-406d-b050-d9d35ea5ba38 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:50.617707645 +0000 UTC m=+36.071513601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls") pod "image-registry-6468cc4b89-rwvgm" (UID: "7c52ab3a-81f5-406d-b050-d9d35ea5ba38") : secret "image-registry-tls" not found Sep 11 18:48:50.117801 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.117747 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzkmr\" (UniqueName: \"kubernetes.io/projected/ad0303b1-5952-4ffc-b1d1-d256b97ba1e6-kube-api-access-gzkmr\") pod \"ingress-canary-g7ldv\" (UID: \"ad0303b1-5952-4ffc-b1d1-d256b97ba1e6\") " pod="openshift-ingress-canary/ingress-canary-g7ldv" Sep 11 18:48:50.117801 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.117779 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-installation-pull-secrets\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.117801 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.117803 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad0303b1-5952-4ffc-b1d1-d256b97ba1e6-cert\") pod \"ingress-canary-g7ldv\" (UID: \"ad0303b1-5952-4ffc-b1d1-d256b97ba1e6\") " pod="openshift-ingress-canary/ingress-canary-g7ldv" Sep 11 18:48:50.118433 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.117825 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bebc8327-abaa-4039-96f8-90928ef2e27d-tmp-dir\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:48:50.118433 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:50.118055 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:48:50.118433 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.118095 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-ca-trust-extracted\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.118433 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:50.118111 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls podName:bebc8327-abaa-4039-96f8-90928ef2e27d nodeName:}" failed. No retries permitted until 2026-09-11 18:48:50.618093004 +0000 UTC m=+36.071898944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls") pod "dns-default-h928z" (UID: "bebc8327-abaa-4039-96f8-90928ef2e27d") : secret "dns-default-metrics-tls" not found Sep 11 18:48:50.118433 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.118121 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bebc8327-abaa-4039-96f8-90928ef2e27d-tmp-dir\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:48:50.118433 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.118279 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bebc8327-abaa-4039-96f8-90928ef2e27d-config-volume\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:48:50.118433 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.118340 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-certificates\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.118927 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.118906 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-trusted-ca\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.122142 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.122118 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-image-registry-private-configuration\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.122142 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.122126 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad0303b1-5952-4ffc-b1d1-d256b97ba1e6-cert\") pod \"ingress-canary-g7ldv\" (UID: \"ad0303b1-5952-4ffc-b1d1-d256b97ba1e6\") " pod="openshift-ingress-canary/ingress-canary-g7ldv" Sep 11 18:48:50.127109 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.127084 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6j6j\" (UniqueName: \"kubernetes.io/projected/bebc8327-abaa-4039-96f8-90928ef2e27d-kube-api-access-h6j6j\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:48:50.127226 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.127205 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzkmr\" (UniqueName: \"kubernetes.io/projected/ad0303b1-5952-4ffc-b1d1-d256b97ba1e6-kube-api-access-gzkmr\") pod \"ingress-canary-g7ldv\" (UID: \"ad0303b1-5952-4ffc-b1d1-d256b97ba1e6\") " pod="openshift-ingress-canary/ingress-canary-g7ldv" Sep 11 18:48:50.127393 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.127372 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-459b4\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-kube-api-access-459b4\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.128124 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.128105 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-bound-sa-token\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.131084 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.131065 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-installation-pull-secrets\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.239813 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.239772 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g7ldv" Sep 11 18:48:50.604101 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.604068 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g7ldv"] Sep 11 18:48:50.607831 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:48:50.607808 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0303b1_5952_4ffc_b1d1_d256b97ba1e6.slice/crio-23d754776ba649c0b43a42902d36849ff0dce8be5134905f6d7dd68965809b3d WatchSource:0}: Error finding container 23d754776ba649c0b43a42902d36849ff0dce8be5134905f6d7dd68965809b3d: Status 404 returned error can't find the container with id 23d754776ba649c0b43a42902d36849ff0dce8be5134905f6d7dd68965809b3d Sep 11 18:48:50.622236 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.622215 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:48:50.622367 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:50.622348 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:48:50.622432 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:50.622392 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:50.622432 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:50.622402 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls podName:bebc8327-abaa-4039-96f8-90928ef2e27d nodeName:}" failed. No retries permitted until 2026-09-11 18:48:51.622387861 +0000 UTC m=+37.076193810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls") pod "dns-default-h928z" (UID: "bebc8327-abaa-4039-96f8-90928ef2e27d") : secret "dns-default-metrics-tls" not found Sep 11 18:48:50.622567 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:50.622493 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 18:48:50.622567 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:50.622503 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6468cc4b89-rwvgm: secret "image-registry-tls" not found Sep 11 18:48:50.622567 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:50.622560 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls podName:7c52ab3a-81f5-406d-b050-d9d35ea5ba38 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:51.622533926 +0000 UTC m=+37.076339860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls") pod "image-registry-6468cc4b89-rwvgm" (UID: "7c52ab3a-81f5-406d-b050-d9d35ea5ba38") : secret "image-registry-tls" not found Sep 11 18:48:51.165594 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:51.165563 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:48:51.166193 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:51.165606 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:48:51.166193 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:51.165562 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:48:51.168033 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:51.168008 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 18:48:51.168850 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:51.168714 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kzbqc\"" Sep 11 18:48:51.168850 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:51.168718 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tr6q7\"" Sep 11 18:48:51.168850 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:51.168768 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 18:48:51.168850 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:51.168775 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 18:48:51.168850 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:51.168722 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 18:48:51.441670 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:51.441589 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g7ldv" event={"ID":"ad0303b1-5952-4ffc-b1d1-d256b97ba1e6","Type":"ContainerStarted","Data":"23d754776ba649c0b43a42902d36849ff0dce8be5134905f6d7dd68965809b3d"} Sep 11 18:48:51.444814 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:51.444786 2556 generic.go:356] "Generic (PLEG): container finished" podID="2fb6ab28-29cd-45d4-8057-68bc5875c32e" containerID="ad20fbdf0881151a3493be95fa2700ef0d02ecd0d07a7b4226a040b53c13aa16" exitCode=0 Sep 11 18:48:51.444946 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:51.444851 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lphqn" event={"ID":"2fb6ab28-29cd-45d4-8057-68bc5875c32e","Type":"ContainerDied","Data":"ad20fbdf0881151a3493be95fa2700ef0d02ecd0d07a7b4226a040b53c13aa16"} Sep 11 18:48:51.631864 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:51.631836 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:51.631972 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:51.631885 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:48:51.632037 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:51.631991 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:48:51.632092 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:51.632043 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls podName:bebc8327-abaa-4039-96f8-90928ef2e27d nodeName:}" failed. No retries permitted until 2026-09-11 18:48:53.632027243 +0000 UTC m=+39.085833177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls") pod "dns-default-h928z" (UID: "bebc8327-abaa-4039-96f8-90928ef2e27d") : secret "dns-default-metrics-tls" not found Sep 11 18:48:51.632092 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:51.631994 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 18:48:51.632092 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:51.632083 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6468cc4b89-rwvgm: secret "image-registry-tls" not found Sep 11 18:48:51.632239 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:51.632124 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls podName:7c52ab3a-81f5-406d-b050-d9d35ea5ba38 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:53.632112887 +0000 UTC m=+39.085918835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls") pod "image-registry-6468cc4b89-rwvgm" (UID: "7c52ab3a-81f5-406d-b050-d9d35ea5ba38") : secret "image-registry-tls" not found Sep 11 18:48:52.450940 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:52.450900 2556 generic.go:356] "Generic (PLEG): container finished" podID="2fb6ab28-29cd-45d4-8057-68bc5875c32e" containerID="f5e96e0b38ebe26a27a328fafa0930821b989285c721e594cc3d1c5ba6d4749b" exitCode=0 Sep 11 18:48:52.451524 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:52.450954 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lphqn" event={"ID":"2fb6ab28-29cd-45d4-8057-68bc5875c32e","Type":"ContainerDied","Data":"f5e96e0b38ebe26a27a328fafa0930821b989285c721e594cc3d1c5ba6d4749b"} Sep 11 18:48:53.455050 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:53.454956 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g7ldv" event={"ID":"ad0303b1-5952-4ffc-b1d1-d256b97ba1e6","Type":"ContainerStarted","Data":"c2811c9a4cc71749c0771f79565b1e3dba9bd796730d3fcc702906acf61caf11"} Sep 11 18:48:53.458276 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:53.458253 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lphqn" event={"ID":"2fb6ab28-29cd-45d4-8057-68bc5875c32e","Type":"ContainerStarted","Data":"c747a4a0bb5ded61a925a910e9868feaf1e1d8d56074be5476b9a64aef8a4c1f"} Sep 11 18:48:53.470272 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:53.470232 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-g7ldv" podStartSLOduration=1.945390607 podStartE2EDuration="4.470218814s" totalImagesPullingTime="2.524828207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:53.469320127 +0000 UTC m=+38.923126082" watchObservedRunningTime="2026-09-11 18:48:53.470218814 +0000 UTC m=+38.924024763" Sep 11 18:48:53.488392 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:53.488347 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lphqn" podStartSLOduration=8.88843688 podStartE2EDuration="38.488332769s" totalImagesPullingTime="29.599895889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:53.487416638 +0000 UTC m=+38.941222618" watchObservedRunningTime="2026-09-11 18:48:53.488332769 +0000 UTC m=+38.942138728" Sep 11 18:48:53.641917 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:53.641872 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:48:53.647860 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:53.647837 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:53.647928 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:53.647872 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:48:53.647986 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:53.647964 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:48:53.648111 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:53.647969 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 18:48:53.648111 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:53.648016 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6468cc4b89-rwvgm: secret "image-registry-tls" not found Sep 11 18:48:53.648111 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:53.648019 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls podName:bebc8327-abaa-4039-96f8-90928ef2e27d nodeName:}" failed. No retries permitted until 2026-09-11 18:48:57.648003302 +0000 UTC m=+43.101809234 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls") pod "dns-default-h928z" (UID: "bebc8327-abaa-4039-96f8-90928ef2e27d") : secret "dns-default-metrics-tls" not found Sep 11 18:48:53.648111 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:53.648057 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls podName:7c52ab3a-81f5-406d-b050-d9d35ea5ba38 nodeName:}" failed. No retries permitted until 2026-09-11 18:48:57.648044475 +0000 UTC m=+43.101850414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls") pod "image-registry-6468cc4b89-rwvgm" (UID: "7c52ab3a-81f5-406d-b050-d9d35ea5ba38") : secret "image-registry-tls" not found Sep 11 18:48:53.753057 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:53.752971 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:48:54.223369 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:54.223338 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:48:54.298573 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:54.298517 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:48:57.677296 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:57.677245 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:48:57.677705 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:48:57.677304 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:48:57.677705 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:57.677406 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:48:57.677705 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:57.677409 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 18:48:57.677705 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:57.677427 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6468cc4b89-rwvgm: secret "image-registry-tls" not found Sep 11 18:48:57.677705 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:57.677472 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls podName:7c52ab3a-81f5-406d-b050-d9d35ea5ba38 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:05.677453706 +0000 UTC m=+51.131259651 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls") pod "image-registry-6468cc4b89-rwvgm" (UID: "7c52ab3a-81f5-406d-b050-d9d35ea5ba38") : secret "image-registry-tls" not found Sep 11 18:48:57.677705 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:48:57.677487 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls podName:bebc8327-abaa-4039-96f8-90928ef2e27d nodeName:}" failed. No retries permitted until 2026-09-11 18:49:05.67747954 +0000 UTC m=+51.131285473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls") pod "dns-default-h928z" (UID: "bebc8327-abaa-4039-96f8-90928ef2e27d") : secret "dns-default-metrics-tls" not found Sep 11 18:49:01.288384 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:01.288347 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:01.354287 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:01.354246 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:04.255625 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:04.255585 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:04.355325 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:04.355289 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:04.446011 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:04.445977 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:05.403521 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:05.403487 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:05.470261 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:05.470212 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:05.539013 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:05.538974 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:05.736518 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:05.736430 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:49:05.736518 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:05.736472 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:49:05.736717 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:05.736589 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:49:05.736717 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:05.736595 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 18:49:05.736717 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:05.736614 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6468cc4b89-rwvgm: secret "image-registry-tls" not found Sep 11 18:49:05.736717 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:05.736643 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls podName:bebc8327-abaa-4039-96f8-90928ef2e27d nodeName:}" failed. No retries permitted until 2026-09-11 18:49:21.736628943 +0000 UTC m=+67.190434876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls") pod "dns-default-h928z" (UID: "bebc8327-abaa-4039-96f8-90928ef2e27d") : secret "dns-default-metrics-tls" not found Sep 11 18:49:05.736717 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:05.736657 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls podName:7c52ab3a-81f5-406d-b050-d9d35ea5ba38 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:21.736650892 +0000 UTC m=+67.190456824 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls") pod "image-registry-6468cc4b89-rwvgm" (UID: "7c52ab3a-81f5-406d-b050-d9d35ea5ba38") : secret "image-registry-tls" not found Sep 11 18:49:06.284013 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:06.283972 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:06.715134 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:06.715043 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:06.824258 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:06.824223 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:06.922421 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:06.922373 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:07.653417 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:07.653353 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:07.751955 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:07.751917 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:11.669718 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:11.669684 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:14.431006 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:14.430970 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-42nvj" Sep 11 18:49:14.592714 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:14.592671 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:14.679187 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:14.679141 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:16.982340 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:16.982270 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:17.084662 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:17.084623 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:18.400260 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:18.400225 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:18.493885 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:18.493836 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:19.967415 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:19.967378 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:20.066176 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:20.066148 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:20.741847 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:20.741809 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret\") pod \"global-pull-secret-syncer-r2xrg\" (UID: \"953eaca9-ab99-4bce-b95a-ab3d3fbad183\") " pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:49:20.742046 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:20.741858 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs\") pod \"network-metrics-daemon-rhmpv\" (UID: \"681500e4-89ea-42a9-9d91-0d44e7698747\") " pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:49:20.744206 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:20.744183 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 18:49:20.744342 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:20.744325 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 18:49:20.752749 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:20.752730 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 18:49:20.752827 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:20.752785 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs podName:681500e4-89ea-42a9-9d91-0d44e7698747 nodeName:}" failed. No retries permitted until 2026-09-11 18:50:24.7527704 +0000 UTC m=+130.206576334 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs") pod "network-metrics-daemon-rhmpv" (UID: "681500e4-89ea-42a9-9d91-0d44e7698747") : secret "metrics-daemon-secret" not found Sep 11 18:49:20.755820 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:20.755789 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/953eaca9-ab99-4bce-b95a-ab3d3fbad183-original-pull-secret\") pod \"global-pull-secret-syncer-r2xrg\" (UID: \"953eaca9-ab99-4bce-b95a-ab3d3fbad183\") " pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:49:20.842794 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:20.842767 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtg7v\" (UniqueName: \"kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v\") pod \"network-check-target-grcgm\" (UID: \"fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0\") " pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:49:20.845157 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:20.845141 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 18:49:20.855068 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:20.855051 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 18:49:20.866071 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:20.866045 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtg7v\" (UniqueName: \"kubernetes.io/projected/fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0-kube-api-access-mtg7v\") pod \"network-check-target-grcgm\" (UID: \"fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0\") " pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:49:20.880023 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:20.879999 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tr6q7\"" Sep 11 18:49:20.888936 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:20.888920 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:49:20.891200 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:20.891175 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2xrg" Sep 11 18:49:21.159784 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:21.159751 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r2xrg"] Sep 11 18:49:21.162411 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:21.162387 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-grcgm"] Sep 11 18:49:21.163388 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:49:21.163359 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953eaca9_ab99_4bce_b95a_ab3d3fbad183.slice/crio-d50d3dc128d85af6ce86f9cfd6cf28bab5ca11de8a2f8582bb9e762f2f676628 WatchSource:0}: Error finding container d50d3dc128d85af6ce86f9cfd6cf28bab5ca11de8a2f8582bb9e762f2f676628: Status 404 returned error can't find the container with id d50d3dc128d85af6ce86f9cfd6cf28bab5ca11de8a2f8582bb9e762f2f676628 Sep 11 18:49:21.165608 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:49:21.165582 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8f4c4c_1358_4ed8_8108_0f87c4a1aaf0.slice/crio-02664dd117b8432f5e88951cc3a85b28afbd3ef54cdcd13e6824893cc32d5093 WatchSource:0}: Error finding container 02664dd117b8432f5e88951cc3a85b28afbd3ef54cdcd13e6824893cc32d5093: Status 404 returned error can't find the container with id 02664dd117b8432f5e88951cc3a85b28afbd3ef54cdcd13e6824893cc32d5093 Sep 11 18:49:21.414931 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:21.414857 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:21.481506 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:21.481449 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:21.525306 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:21.525275 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-grcgm" event={"ID":"fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0","Type":"ContainerStarted","Data":"02664dd117b8432f5e88951cc3a85b28afbd3ef54cdcd13e6824893cc32d5093"} Sep 11 18:49:21.526306 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:21.526280 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r2xrg" event={"ID":"953eaca9-ab99-4bce-b95a-ab3d3fbad183","Type":"ContainerStarted","Data":"d50d3dc128d85af6ce86f9cfd6cf28bab5ca11de8a2f8582bb9e762f2f676628"} Sep 11 18:49:21.750389 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:21.750296 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:49:21.750389 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:21.750350 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:49:21.750635 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:21.750463 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 18:49:21.750635 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:21.750467 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 18:49:21.750635 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:21.750488 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6468cc4b89-rwvgm: secret "image-registry-tls" not found Sep 11 18:49:21.750635 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:21.750522 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls podName:bebc8327-abaa-4039-96f8-90928ef2e27d nodeName:}" failed. No retries permitted until 2026-09-11 18:49:53.750504519 +0000 UTC m=+99.204310456 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls") pod "dns-default-h928z" (UID: "bebc8327-abaa-4039-96f8-90928ef2e27d") : secret "dns-default-metrics-tls" not found Sep 11 18:49:21.750635 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:21.750558 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls podName:7c52ab3a-81f5-406d-b050-d9d35ea5ba38 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:53.750531038 +0000 UTC m=+99.204336977 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls") pod "image-registry-6468cc4b89-rwvgm" (UID: "7c52ab3a-81f5-406d-b050-d9d35ea5ba38") : secret "image-registry-tls" not found Sep 11 18:49:21.976442 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:21.976404 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll"] Sep 11 18:49:21.978464 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:21.978441 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll" Sep 11 18:49:21.980598 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:21.980576 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 11 18:49:21.980697 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:21.980576 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 11 18:49:21.980697 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:21.980584 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kbtqs\"" Sep 11 18:49:21.981148 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:21.981130 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:49:21.981230 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:21.981166 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 11 18:49:21.991011 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:21.990989 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll"] Sep 11 18:49:22.052334 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.052303 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12-serving-cert\") pod \"service-ca-operator-cf7c776b7-57sll\" (UID: \"8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll" Sep 11 18:49:22.052505 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.052344 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5lhw\" (UniqueName: \"kubernetes.io/projected/8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12-kube-api-access-p5lhw\") pod \"service-ca-operator-cf7c776b7-57sll\" (UID: \"8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll" Sep 11 18:49:22.052505 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.052377 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12-config\") pod \"service-ca-operator-cf7c776b7-57sll\" (UID: \"8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll" Sep 11 18:49:22.084853 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.084824 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-kxkhf"] Sep 11 18:49:22.086957 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.086930 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.090869 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.090776 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 11 18:49:22.091363 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.091304 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bqznc\"" Sep 11 18:49:22.091363 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.090991 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 18:49:22.091514 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.090827 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 11 18:49:22.091748 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.091678 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 18:49:22.097747 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.097568 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 11 18:49:22.102082 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.102063 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-kxkhf"] Sep 11 18:49:22.152865 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.152839 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12-serving-cert\") pod \"service-ca-operator-cf7c776b7-57sll\" (UID: \"8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll" Sep 11 18:49:22.152865 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.152870 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5lhw\" (UniqueName: \"kubernetes.io/projected/8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12-kube-api-access-p5lhw\") pod \"service-ca-operator-cf7c776b7-57sll\" (UID: \"8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll" Sep 11 18:49:22.153059 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.152892 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9462a6cd-3ef7-4962-aaea-01017510998d-service-ca-bundle\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.153059 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.152912 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12-config\") pod \"service-ca-operator-cf7c776b7-57sll\" (UID: \"8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll" Sep 11 18:49:22.153059 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.152983 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9462a6cd-3ef7-4962-aaea-01017510998d-tmp\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.153059 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.153024 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9462a6cd-3ef7-4962-aaea-01017510998d-trusted-ca-bundle\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.153059 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.153048 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4mks\" (UniqueName: \"kubernetes.io/projected/9462a6cd-3ef7-4962-aaea-01017510998d-kube-api-access-z4mks\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.153308 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.153079 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9462a6cd-3ef7-4962-aaea-01017510998d-snapshots\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.153308 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.153106 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9462a6cd-3ef7-4962-aaea-01017510998d-serving-cert\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.153720 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.153674 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12-config\") pod \"service-ca-operator-cf7c776b7-57sll\" (UID: \"8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll" Sep 11 18:49:22.156015 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.155971 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12-serving-cert\") pod \"service-ca-operator-cf7c776b7-57sll\" (UID: \"8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll" Sep 11 18:49:22.162387 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.162360 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5lhw\" (UniqueName: \"kubernetes.io/projected/8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12-kube-api-access-p5lhw\") pod \"service-ca-operator-cf7c776b7-57sll\" (UID: \"8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll" Sep 11 18:49:22.177922 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.177891 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk"] Sep 11 18:49:22.180103 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.180080 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk" Sep 11 18:49:22.182714 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.182677 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 11 18:49:22.183239 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.183130 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-glk7z\"" Sep 11 18:49:22.183347 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.183248 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 11 18:49:22.183469 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.183451 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 11 18:49:22.183805 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.183784 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:49:22.192736 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.192714 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk"] Sep 11 18:49:22.254025 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.253991 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9462a6cd-3ef7-4962-aaea-01017510998d-trusted-ca-bundle\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.254025 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.254027 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4mks\" (UniqueName: \"kubernetes.io/projected/9462a6cd-3ef7-4962-aaea-01017510998d-kube-api-access-z4mks\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.254253 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.254049 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9462a6cd-3ef7-4962-aaea-01017510998d-snapshots\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.254253 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.254095 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-m2xrk\" (UID: \"8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk" Sep 11 18:49:22.254253 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.254126 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9462a6cd-3ef7-4962-aaea-01017510998d-serving-cert\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.254253 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.254149 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9462a6cd-3ef7-4962-aaea-01017510998d-service-ca-bundle\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.254253 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.254169 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-m2xrk\" (UID: \"8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk" Sep 11 18:49:22.254253 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.254193 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkcr6\" (UniqueName: \"kubernetes.io/projected/8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1-kube-api-access-pkcr6\") pod \"kube-storage-version-migrator-operator-6c8f84f897-m2xrk\" (UID: \"8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk" Sep 11 18:49:22.254583 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.254261 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9462a6cd-3ef7-4962-aaea-01017510998d-tmp\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.254819 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.254795 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9462a6cd-3ef7-4962-aaea-01017510998d-tmp\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.255031 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.255005 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9462a6cd-3ef7-4962-aaea-01017510998d-trusted-ca-bundle\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.255204 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.255182 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9462a6cd-3ef7-4962-aaea-01017510998d-service-ca-bundle\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.255280 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.255241 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9462a6cd-3ef7-4962-aaea-01017510998d-snapshots\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.257074 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.257050 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9462a6cd-3ef7-4962-aaea-01017510998d-serving-cert\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.268039 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.268016 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4mks\" (UniqueName: \"kubernetes.io/projected/9462a6cd-3ef7-4962-aaea-01017510998d-kube-api-access-z4mks\") pod \"insights-operator-5844b856db-kxkhf\" (UID: \"9462a6cd-3ef7-4962-aaea-01017510998d\") " pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.290753 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.290724 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll" Sep 11 18:49:22.310267 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.310205 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:22.355117 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.354963 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-m2xrk\" (UID: \"8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk" Sep 11 18:49:22.355117 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.355011 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-m2xrk\" (UID: \"8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk" Sep 11 18:49:22.355117 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.355043 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkcr6\" (UniqueName: \"kubernetes.io/projected/8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1-kube-api-access-pkcr6\") pod \"kube-storage-version-migrator-operator-6c8f84f897-m2xrk\" (UID: \"8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk" Sep 11 18:49:22.355637 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.355573 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-m2xrk\" (UID: \"8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk" Sep 11 18:49:22.357481 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.357406 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-m2xrk\" (UID: \"8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk" Sep 11 18:49:22.362977 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.362943 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkcr6\" (UniqueName: \"kubernetes.io/projected/8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1-kube-api-access-pkcr6\") pod \"kube-storage-version-migrator-operator-6c8f84f897-m2xrk\" (UID: \"8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk" Sep 11 18:49:22.398821 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.398784 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-kxkhf" Sep 11 18:49:22.413231 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.413177 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:22.492046 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.491987 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk" Sep 11 18:49:22.511973 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.511947 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:22.569029 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.568873 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll"] Sep 11 18:49:22.573493 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:49:22.573465 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8dd4b3_8dfa_4b38_9d18_70f2187fdb12.slice/crio-eedcf88ef711f0d61eb9bd6da2f57ff51e7d16f0727a4a69ba4a41e49618f6d9 WatchSource:0}: Error finding container eedcf88ef711f0d61eb9bd6da2f57ff51e7d16f0727a4a69ba4a41e49618f6d9: Status 404 returned error can't find the container with id eedcf88ef711f0d61eb9bd6da2f57ff51e7d16f0727a4a69ba4a41e49618f6d9 Sep 11 18:49:22.697433 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.697354 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-kxkhf"] Sep 11 18:49:22.700454 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:49:22.700422 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9462a6cd_3ef7_4962_aaea_01017510998d.slice/crio-2fc5310ff3d87a95fa8dc525b6b4e38c1228c6ae3d058a2660be74b1e596e873 WatchSource:0}: Error finding container 2fc5310ff3d87a95fa8dc525b6b4e38c1228c6ae3d058a2660be74b1e596e873: Status 404 returned error can't find the container with id 2fc5310ff3d87a95fa8dc525b6b4e38c1228c6ae3d058a2660be74b1e596e873 Sep 11 18:49:22.770437 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:22.770406 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk"] Sep 11 18:49:22.774613 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:49:22.774577 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7101b1_479d_4ee5_9d3b_4df6a63dc9d1.slice/crio-340a3ca94aa4e9780348725cafbcd07f2890c3a6971c93d6cb4782124d7fe65c WatchSource:0}: Error finding container 340a3ca94aa4e9780348725cafbcd07f2890c3a6971c93d6cb4782124d7fe65c: Status 404 returned error can't find the container with id 340a3ca94aa4e9780348725cafbcd07f2890c3a6971c93d6cb4782124d7fe65c Sep 11 18:49:23.261364 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:23.261328 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:23.336520 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:23.336459 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:23.539375 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:23.539259 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-kxkhf" event={"ID":"9462a6cd-3ef7-4962-aaea-01017510998d","Type":"ContainerStarted","Data":"2fc5310ff3d87a95fa8dc525b6b4e38c1228c6ae3d058a2660be74b1e596e873"} Sep 11 18:49:23.542473 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:23.542398 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk" event={"ID":"8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1","Type":"ContainerStarted","Data":"340a3ca94aa4e9780348725cafbcd07f2890c3a6971c93d6cb4782124d7fe65c"} Sep 11 18:49:23.544877 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:23.544806 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll" event={"ID":"8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12","Type":"ContainerStarted","Data":"eedcf88ef711f0d61eb9bd6da2f57ff51e7d16f0727a4a69ba4a41e49618f6d9"} Sep 11 18:49:23.659112 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:23.659077 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:23.762951 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:23.762739 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:23.906373 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:23.906177 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:24.008234 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:24.008175 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:25.737051 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:25.737016 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:25.838568 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:25.838511 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:28.560996 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:28.560959 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r2xrg" event={"ID":"953eaca9-ab99-4bce-b95a-ab3d3fbad183","Type":"ContainerStarted","Data":"9025876c6cd8421234942854dbdec68047acf386bfcef1a99c58decae092f555"} Sep 11 18:49:28.563066 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:28.563034 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-kxkhf" event={"ID":"9462a6cd-3ef7-4962-aaea-01017510998d","Type":"ContainerStarted","Data":"fd64c1875ad870b871b16c1ce664e7155bc373a52b92f1240292e23b4f1e7077"} Sep 11 18:49:28.564601 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:28.564574 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-grcgm" event={"ID":"fe8f4c4c-1358-4ed8-8108-0f87c4a1aaf0","Type":"ContainerStarted","Data":"fcd3434141ace5f19c63dc33a280147be94e187e9ead25dea52b1672407a023a"} Sep 11 18:49:28.564757 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:28.564680 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:49:28.566091 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:28.566065 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk" event={"ID":"8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1","Type":"ContainerStarted","Data":"e1b1a766058e453e25771e5397ee0e8896562897ac5e3c2c31e0bb51d21dfa33"} Sep 11 18:49:28.567500 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:28.567480 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll" event={"ID":"8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12","Type":"ContainerStarted","Data":"c38cd56bf4efb32c1c73a07811f54ac3c394c9324e2ce9f746f97c75e6e22419"} Sep 11 18:49:28.576817 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:28.576777 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-r2xrg" podStartSLOduration=66.621962365 podStartE2EDuration="1m13.576766303s" totalImagesPullingTime="6.954803938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:28.576626428 +0000 UTC m=+74.030432404" watchObservedRunningTime="2026-09-11 18:49:28.576766303 +0000 UTC m=+74.030572249" Sep 11 18:49:28.595450 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:28.595412 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-kxkhf" podStartSLOduration=1.190699227 podStartE2EDuration="6.595400081s" totalImagesPullingTime="5.404700854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:28.59468057 +0000 UTC m=+74.048486539" watchObservedRunningTime="2026-09-11 18:49:28.595400081 +0000 UTC m=+74.049206036" Sep 11 18:49:28.611438 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:28.611162 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk" podStartSLOduration=1.28072626 podStartE2EDuration="6.611146422s" totalImagesPullingTime="5.330420162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:28.610603207 +0000 UTC m=+74.064409187" watchObservedRunningTime="2026-09-11 18:49:28.611146422 +0000 UTC m=+74.064952378" Sep 11 18:49:28.629858 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:28.629818 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-grcgm" podStartSLOduration=66.690152266 podStartE2EDuration="1m13.6298041s" totalImagesPullingTime="6.939651834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:28.628484726 +0000 UTC m=+74.082290664" watchObservedRunningTime="2026-09-11 18:49:28.6298041 +0000 UTC m=+74.083610056" Sep 11 18:49:28.646839 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:28.646797 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll" podStartSLOduration=2.108963493 podStartE2EDuration="7.646781643s" totalImagesPullingTime="5.53781815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:28.645788833 +0000 UTC m=+74.099594789" watchObservedRunningTime="2026-09-11 18:49:28.646781643 +0000 UTC m=+74.100587599" Sep 11 18:49:29.709124 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:29.709086 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:29.744508 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:29.744477 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-tpxz7"] Sep 11 18:49:29.748004 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:29.747981 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tpxz7" Sep 11 18:49:29.750512 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:29.750492 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ffb5r\"" Sep 11 18:49:29.751310 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:29.751283 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 11 18:49:29.751430 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:29.751332 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 11 18:49:29.759236 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:29.759215 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-tpxz7"] Sep 11 18:49:29.799160 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:29.799130 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:29.823328 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:29.823303 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87zgr\" (UniqueName: \"kubernetes.io/projected/d3d4ec18-eacd-43c7-a4fd-31e078bcc6bf-kube-api-access-87zgr\") pod \"migrator-8b4d697c5-tpxz7\" (UID: \"d3d4ec18-eacd-43c7-a4fd-31e078bcc6bf\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tpxz7" Sep 11 18:49:29.900470 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:29.900434 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:29.924476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:29.924443 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87zgr\" (UniqueName: \"kubernetes.io/projected/d3d4ec18-eacd-43c7-a4fd-31e078bcc6bf-kube-api-access-87zgr\") pod \"migrator-8b4d697c5-tpxz7\" (UID: \"d3d4ec18-eacd-43c7-a4fd-31e078bcc6bf\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tpxz7" Sep 11 18:49:29.932245 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:29.932221 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87zgr\" (UniqueName: \"kubernetes.io/projected/d3d4ec18-eacd-43c7-a4fd-31e078bcc6bf-kube-api-access-87zgr\") pod \"migrator-8b4d697c5-tpxz7\" (UID: \"d3d4ec18-eacd-43c7-a4fd-31e078bcc6bf\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tpxz7" Sep 11 18:49:30.005680 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.005590 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:30.048471 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.048435 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nvjvz"] Sep 11 18:49:30.051438 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.051413 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:30.053830 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.053810 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 18:49:30.054041 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.054017 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 18:49:30.054193 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.054178 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6nt5d\"" Sep 11 18:49:30.056125 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.056107 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tpxz7" Sep 11 18:49:30.067600 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.067580 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nvjvz"] Sep 11 18:49:30.096360 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.096322 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:30.126164 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.126067 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3f67d93-49c4-45a9-803b-806c6446432c-crio-socket\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:30.126367 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.126326 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:30.126926 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.126800 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3f67d93-49c4-45a9-803b-806c6446432c-data-volume\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:30.126926 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.126841 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:30.126926 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.126858 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7nb9\" (UniqueName: \"kubernetes.io/projected/f3f67d93-49c4-45a9-803b-806c6446432c-kube-api-access-m7nb9\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:30.227173 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.227142 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3f67d93-49c4-45a9-803b-806c6446432c-data-volume\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:30.227319 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.227184 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:30.227319 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.227202 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7nb9\" (UniqueName: \"kubernetes.io/projected/f3f67d93-49c4-45a9-803b-806c6446432c-kube-api-access-m7nb9\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:30.227403 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.227291 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3f67d93-49c4-45a9-803b-806c6446432c-crio-socket\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:30.227403 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:30.227323 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 18:49:30.227403 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.227355 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:30.227403 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:30.227388 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls podName:f3f67d93-49c4-45a9-803b-806c6446432c nodeName:}" failed. No retries permitted until 2026-09-11 18:49:30.727374263 +0000 UTC m=+76.181180195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nvjvz" (UID: "f3f67d93-49c4-45a9-803b-806c6446432c") : secret "insights-runtime-extractor-tls" not found Sep 11 18:49:30.227614 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:30.227443 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 18:49:30.227614 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.227465 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3f67d93-49c4-45a9-803b-806c6446432c-crio-socket\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:30.227614 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:30.227504 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm podName:f3f67d93-49c4-45a9-803b-806c6446432c nodeName:}" failed. No retries permitted until 2026-09-11 18:49:30.727492904 +0000 UTC m=+76.181298839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nvjvz" (UID: "f3f67d93-49c4-45a9-803b-806c6446432c") : configmap "kube-rbac-proxy" not found Sep 11 18:49:30.227734 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.227636 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3f67d93-49c4-45a9-803b-806c6446432c-data-volume\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:30.236224 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.236204 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7nb9\" (UniqueName: \"kubernetes.io/projected/f3f67d93-49c4-45a9-803b-806c6446432c-kube-api-access-m7nb9\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:30.305267 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.305240 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:30.331467 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.331444 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-tpxz7"] Sep 11 18:49:30.334753 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:49:30.334733 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d4ec18_eacd_43c7_a4fd_31e078bcc6bf.slice/crio-6cda0cb5894a6cdc16166af2bf14d97cb1d293b6eb07c43f9a7c623ae85644b0 WatchSource:0}: Error finding container 6cda0cb5894a6cdc16166af2bf14d97cb1d293b6eb07c43f9a7c623ae85644b0: Status 404 returned error can't find the container with id 6cda0cb5894a6cdc16166af2bf14d97cb1d293b6eb07c43f9a7c623ae85644b0 Sep 11 18:49:30.405495 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.405470 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:30.575397 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.575322 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tpxz7" event={"ID":"d3d4ec18-eacd-43c7-a4fd-31e078bcc6bf","Type":"ContainerStarted","Data":"6cda0cb5894a6cdc16166af2bf14d97cb1d293b6eb07c43f9a7c623ae85644b0"} Sep 11 18:49:30.579083 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.579061 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:30.679323 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.679295 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:30.730646 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.730619 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:30.730955 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:30.730700 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:30.730955 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:30.730770 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 18:49:30.730955 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:30.730803 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 18:49:30.730955 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:30.730837 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls podName:f3f67d93-49c4-45a9-803b-806c6446432c nodeName:}" failed. No retries permitted until 2026-09-11 18:49:31.730819303 +0000 UTC m=+77.184625256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nvjvz" (UID: "f3f67d93-49c4-45a9-803b-806c6446432c") : secret "insights-runtime-extractor-tls" not found Sep 11 18:49:30.730955 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:30.730859 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm podName:f3f67d93-49c4-45a9-803b-806c6446432c nodeName:}" failed. No retries permitted until 2026-09-11 18:49:31.730846513 +0000 UTC m=+77.184652446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nvjvz" (UID: "f3f67d93-49c4-45a9-803b-806c6446432c") : configmap "kube-rbac-proxy" not found Sep 11 18:49:31.581050 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:31.581018 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tpxz7" event={"ID":"d3d4ec18-eacd-43c7-a4fd-31e078bcc6bf","Type":"ContainerStarted","Data":"558b83aa01935eb26be6d8fc7243be11e399e4d6a0a745f4937cb8b6f65d0bec"} Sep 11 18:49:31.581182 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:31.581080 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tpxz7" event={"ID":"d3d4ec18-eacd-43c7-a4fd-31e078bcc6bf","Type":"ContainerStarted","Data":"b99e9da29f883bf22d0b126966567716f97b675c0639c52e664f50669b33dded"} Sep 11 18:49:31.601286 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:31.601238 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tpxz7" podStartSLOduration=1.533501816 podStartE2EDuration="2.601221954s" totalImagesPullingTime="1.067720138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:31.599341177 +0000 UTC m=+77.053147132" watchObservedRunningTime="2026-09-11 18:49:31.601221954 +0000 UTC m=+77.055027909" Sep 11 18:49:31.702831 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:31.702748 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:31.737904 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:31.737865 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:31.738226 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:31.737950 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:31.738226 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:31.737996 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 18:49:31.738226 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:31.738050 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm podName:f3f67d93-49c4-45a9-803b-806c6446432c nodeName:}" failed. No retries permitted until 2026-09-11 18:49:33.738037765 +0000 UTC m=+79.191843697 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nvjvz" (UID: "f3f67d93-49c4-45a9-803b-806c6446432c") : configmap "kube-rbac-proxy" not found Sep 11 18:49:31.738226 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:31.738099 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 18:49:31.738226 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:31.738157 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls podName:f3f67d93-49c4-45a9-803b-806c6446432c nodeName:}" failed. No retries permitted until 2026-09-11 18:49:33.738143822 +0000 UTC m=+79.191949756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nvjvz" (UID: "f3f67d93-49c4-45a9-803b-806c6446432c") : secret "insights-runtime-extractor-tls" not found Sep 11 18:49:31.820094 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:31.820066 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:32.197111 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.197079 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-mrrnb"] Sep 11 18:49:32.200037 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.200021 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-mrrnb" Sep 11 18:49:32.206138 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.206119 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 11 18:49:32.209662 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.207920 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 11 18:49:32.209662 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.208229 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 11 18:49:32.209662 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.208514 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 11 18:49:32.209662 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.208589 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8c4x2\"" Sep 11 18:49:32.216208 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.216184 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-mrrnb"] Sep 11 18:49:32.242100 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.242075 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2kz4\" (UniqueName: \"kubernetes.io/projected/9a7e1b0a-dc0b-451b-975b-5a57e7bc0578-kube-api-access-r2kz4\") pod \"service-ca-568869fdc5-mrrnb\" (UID: \"9a7e1b0a-dc0b-451b-975b-5a57e7bc0578\") " pod="openshift-service-ca/service-ca-568869fdc5-mrrnb" Sep 11 18:49:32.242182 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.242163 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9a7e1b0a-dc0b-451b-975b-5a57e7bc0578-signing-cabundle\") pod \"service-ca-568869fdc5-mrrnb\" (UID: \"9a7e1b0a-dc0b-451b-975b-5a57e7bc0578\") " pod="openshift-service-ca/service-ca-568869fdc5-mrrnb" Sep 11 18:49:32.242267 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.242222 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9a7e1b0a-dc0b-451b-975b-5a57e7bc0578-signing-key\") pod \"service-ca-568869fdc5-mrrnb\" (UID: \"9a7e1b0a-dc0b-451b-975b-5a57e7bc0578\") " pod="openshift-service-ca/service-ca-568869fdc5-mrrnb" Sep 11 18:49:32.256801 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.256768 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:32.343108 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.343076 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2kz4\" (UniqueName: \"kubernetes.io/projected/9a7e1b0a-dc0b-451b-975b-5a57e7bc0578-kube-api-access-r2kz4\") pod \"service-ca-568869fdc5-mrrnb\" (UID: \"9a7e1b0a-dc0b-451b-975b-5a57e7bc0578\") " pod="openshift-service-ca/service-ca-568869fdc5-mrrnb" Sep 11 18:49:32.343244 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.343231 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9a7e1b0a-dc0b-451b-975b-5a57e7bc0578-signing-cabundle\") pod \"service-ca-568869fdc5-mrrnb\" (UID: \"9a7e1b0a-dc0b-451b-975b-5a57e7bc0578\") " pod="openshift-service-ca/service-ca-568869fdc5-mrrnb" Sep 11 18:49:32.343284 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.343264 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9a7e1b0a-dc0b-451b-975b-5a57e7bc0578-signing-key\") pod \"service-ca-568869fdc5-mrrnb\" (UID: \"9a7e1b0a-dc0b-451b-975b-5a57e7bc0578\") " pod="openshift-service-ca/service-ca-568869fdc5-mrrnb" Sep 11 18:49:32.343900 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.343877 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9a7e1b0a-dc0b-451b-975b-5a57e7bc0578-signing-cabundle\") pod \"service-ca-568869fdc5-mrrnb\" (UID: \"9a7e1b0a-dc0b-451b-975b-5a57e7bc0578\") " pod="openshift-service-ca/service-ca-568869fdc5-mrrnb" Sep 11 18:49:32.345578 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.345554 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9a7e1b0a-dc0b-451b-975b-5a57e7bc0578-signing-key\") pod \"service-ca-568869fdc5-mrrnb\" (UID: \"9a7e1b0a-dc0b-451b-975b-5a57e7bc0578\") " pod="openshift-service-ca/service-ca-568869fdc5-mrrnb" Sep 11 18:49:32.352756 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.352732 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2kz4\" (UniqueName: \"kubernetes.io/projected/9a7e1b0a-dc0b-451b-975b-5a57e7bc0578-kube-api-access-r2kz4\") pod \"service-ca-568869fdc5-mrrnb\" (UID: \"9a7e1b0a-dc0b-451b-975b-5a57e7bc0578\") " pod="openshift-service-ca/service-ca-568869fdc5-mrrnb" Sep 11 18:49:32.511243 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.511162 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-mrrnb" Sep 11 18:49:32.791665 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:32.791637 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-mrrnb"] Sep 11 18:49:32.794994 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:49:32.794965 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7e1b0a_dc0b_451b_975b_5a57e7bc0578.slice/crio-24ac2b2f7c481dc61484da2025f371638e6e63a03ddf58cf98619925dfef5d67 WatchSource:0}: Error finding container 24ac2b2f7c481dc61484da2025f371638e6e63a03ddf58cf98619925dfef5d67: Status 404 returned error can't find the container with id 24ac2b2f7c481dc61484da2025f371638e6e63a03ddf58cf98619925dfef5d67 Sep 11 18:49:33.005475 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:33.005448 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n8gcj_16a827b3-ee19-4675-ab27-8b47231c2ebc/dns-node-resolver/0.log" Sep 11 18:49:33.341828 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:33.341800 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:33.592724 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:33.592644 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-mrrnb" event={"ID":"9a7e1b0a-dc0b-451b-975b-5a57e7bc0578","Type":"ContainerStarted","Data":"2b8ad8cfaae52a999da3a0a1ff3893720efdb30fdbf05cd598a5e81372759291"} Sep 11 18:49:33.592724 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:33.592701 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-mrrnb" event={"ID":"9a7e1b0a-dc0b-451b-975b-5a57e7bc0578","Type":"ContainerStarted","Data":"24ac2b2f7c481dc61484da2025f371638e6e63a03ddf58cf98619925dfef5d67"} Sep 11 18:49:33.613688 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:33.613634 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-mrrnb" podStartSLOduration=1.61361863 podStartE2EDuration="1.61361863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:33.61289773 +0000 UTC m=+79.066703698" watchObservedRunningTime="2026-09-11 18:49:33.61361863 +0000 UTC m=+79.067424585" Sep 11 18:49:33.753531 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:33.753496 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:33.753689 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:33.753600 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 18:49:33.753689 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:33.753618 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:33.753689 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:33.753655 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm podName:f3f67d93-49c4-45a9-803b-806c6446432c nodeName:}" failed. No retries permitted until 2026-09-11 18:49:37.753637848 +0000 UTC m=+83.207443781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nvjvz" (UID: "f3f67d93-49c4-45a9-803b-806c6446432c") : configmap "kube-rbac-proxy" not found Sep 11 18:49:33.753807 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:33.753719 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 18:49:33.753807 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:33.753772 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls podName:f3f67d93-49c4-45a9-803b-806c6446432c nodeName:}" failed. No retries permitted until 2026-09-11 18:49:37.753756724 +0000 UTC m=+83.207562664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nvjvz" (UID: "f3f67d93-49c4-45a9-803b-806c6446432c") : secret "insights-runtime-extractor-tls" not found Sep 11 18:49:33.803872 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:33.803848 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vm7pc_908b068d-8d94-4fcb-8490-eb8b1ac59a90/node-ca/0.log" Sep 11 18:49:34.506499 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:34.506466 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:34.612034 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:34.611998 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:34.751210 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:34.751181 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:34.800775 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:34.800749 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g7ldv_ad0303b1-5952-4ffc-b1d1-d256b97ba1e6/serve-healthcheck-canary/0.log" Sep 11 18:49:34.855216 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:34.855174 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:34.955397 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:34.955334 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:35.001869 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:35.001847 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-tpxz7_d3d4ec18-eacd-43c7-a4fd-31e078bcc6bf/migrator/0.log" Sep 11 18:49:35.190003 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:35.189919 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:35.229244 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:35.229224 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-tpxz7_d3d4ec18-eacd-43c7-a4fd-31e078bcc6bf/graceful-termination/0.log" Sep 11 18:49:35.403824 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:35.403794 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-m2xrk_8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1/kube-storage-version-migrator-operator/0.log" Sep 11 18:49:35.935730 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:35.935698 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:36.337105 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:36.337076 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:36.749719 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:36.749639 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:36.953641 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:36.953607 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:37.529448 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:37.529322 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:37.630944 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:37.630915 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:37.782906 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:37.782881 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:37.783039 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:37.782937 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:37.783039 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:37.782995 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 18:49:37.783117 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:37.783047 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 18:49:37.783117 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:37.783056 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm podName:f3f67d93-49c4-45a9-803b-806c6446432c nodeName:}" failed. No retries permitted until 2026-09-11 18:49:45.783039588 +0000 UTC m=+91.236845526 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nvjvz" (UID: "f3f67d93-49c4-45a9-803b-806c6446432c") : configmap "kube-rbac-proxy" not found Sep 11 18:49:37.783117 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:37.783090 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls podName:f3f67d93-49c4-45a9-803b-806c6446432c nodeName:}" failed. No retries permitted until 2026-09-11 18:49:45.783078717 +0000 UTC m=+91.236884650 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nvjvz" (UID: "f3f67d93-49c4-45a9-803b-806c6446432c") : secret "insights-runtime-extractor-tls" not found Sep 11 18:49:37.935890 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:37.935846 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:38.134928 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:38.134855 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:38.207464 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:38.207435 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:38.528969 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:38.528865 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:38.758497 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:38.758460 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:38.936973 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:38.936932 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:39.029178 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:39.029155 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:39.347434 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:39.347406 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:40.126098 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:40.126069 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:40.348368 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:40.348328 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:40.723793 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:40.723757 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:40.825640 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:40.825615 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:40.937895 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:40.937852 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:41.329073 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:41.329039 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:41.928550 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:41.928500 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:42.343857 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:42.343810 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:42.729668 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:42.729592 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:42.877858 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:42.877827 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:43.009631 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:43.009533 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:43.344877 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:43.344846 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:44.146763 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:44.146735 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:44.423593 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:44.423515 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:44.752094 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:44.752013 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:44.929627 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:44.929595 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:45.138039 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:45.138008 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:45.236482 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:45.236448 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:45.845110 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:45.845077 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:45.845288 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:45.845137 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:45.845613 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:45.845596 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3f67d93-49c4-45a9-803b-806c6446432c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:45.847407 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:45.847389 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3f67d93-49c4-45a9-803b-806c6446432c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nvjvz\" (UID: \"f3f67d93-49c4-45a9-803b-806c6446432c\") " pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:45.960561 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:45.960522 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nvjvz" Sep 11 18:49:46.233570 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:46.233487 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nvjvz"] Sep 11 18:49:46.246409 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:49:46.246384 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f67d93_49c4_45a9_803b_806c6446432c.slice/crio-f085e26591137bfe42b54f5f18c1e141226ec780d42db0940f36790c5094d18c WatchSource:0}: Error finding container f085e26591137bfe42b54f5f18c1e141226ec780d42db0940f36790c5094d18c: Status 404 returned error can't find the container with id f085e26591137bfe42b54f5f18c1e141226ec780d42db0940f36790c5094d18c Sep 11 18:49:46.306481 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:46.306449 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:46.410598 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:46.410569 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:46.533757 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:46.533724 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:46.611333 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:46.611307 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:46.637383 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:46.637352 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nvjvz" event={"ID":"f3f67d93-49c4-45a9-803b-806c6446432c","Type":"ContainerStarted","Data":"18875d37531b9df127ac9b8c114a90239c9d0cd01f4f17483ebaf570c986a421"} Sep 11 18:49:46.637505 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:46.637427 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nvjvz" event={"ID":"f3f67d93-49c4-45a9-803b-806c6446432c","Type":"ContainerStarted","Data":"f085e26591137bfe42b54f5f18c1e141226ec780d42db0940f36790c5094d18c"} Sep 11 18:49:46.696582 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:46.696532 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:46.929945 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:46.929921 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:47.046187 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:47.046148 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:47.153403 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:47.153330 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:47.294479 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:47.294441 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:47.433696 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:47.433618 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:47.642933 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:47.642898 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nvjvz" event={"ID":"f3f67d93-49c4-45a9-803b-806c6446432c","Type":"ContainerStarted","Data":"bc503acfcf957375e949865b7278bfe2e8cc0db8bc55a57db366e9a86649053c"} Sep 11 18:49:48.100350 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:48.100322 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:48.285495 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:48.285462 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:48.514476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:48.514401 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:48.648076 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:48.648041 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nvjvz" event={"ID":"f3f67d93-49c4-45a9-803b-806c6446432c","Type":"ContainerStarted","Data":"f37f472342ad3cc89fb760db6a1ec963f1aae956e8e257fac54d3163c6acf273"} Sep 11 18:49:48.673385 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:48.673346 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nvjvz" podStartSLOduration=16.963544985 podStartE2EDuration="18.67332911s" totalImagesPullingTime="1.709784125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:49:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:48.673290179 +0000 UTC m=+94.127096151" watchObservedRunningTime="2026-09-11 18:49:48.67332911 +0000 UTC m=+94.127135065" Sep 11 18:49:48.906069 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:48.906037 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:49.012382 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:49.012349 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:49.139746 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:49.139716 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:53.796886 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:53.796841 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:49:53.797246 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:53.796960 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:49:53.799230 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:53.799201 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls\") pod \"image-registry-6468cc4b89-rwvgm\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:49:53.799230 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:53.799205 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bebc8327-abaa-4039-96f8-90928ef2e27d-metrics-tls\") pod \"dns-default-h928z\" (UID: \"bebc8327-abaa-4039-96f8-90928ef2e27d\") " pod="openshift-dns/dns-default-h928z" Sep 11 18:49:53.813926 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:53.813906 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pqdcd\"" Sep 11 18:49:53.818960 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:53.818942 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:49:53.820976 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:53.820960 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vlxp2\"" Sep 11 18:49:53.829224 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:53.829206 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h928z" Sep 11 18:49:54.098972 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:54.098898 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6468cc4b89-rwvgm"] Sep 11 18:49:54.102369 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:49:54.102342 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c52ab3a_81f5_406d_b050_d9d35ea5ba38.slice/crio-d6affd9d22a1c6b49a0be265399e1ccaec97727d1a711eeee507e3313738799f WatchSource:0}: Error finding container d6affd9d22a1c6b49a0be265399e1ccaec97727d1a711eeee507e3313738799f: Status 404 returned error can't find the container with id d6affd9d22a1c6b49a0be265399e1ccaec97727d1a711eeee507e3313738799f Sep 11 18:49:54.112489 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:54.112428 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h928z"] Sep 11 18:49:54.115472 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:49:54.115448 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbebc8327_abaa_4039_96f8_90928ef2e27d.slice/crio-3ff54e224ce5d8b032ae7003d514a61bb74fc004507815319ccc1fcbdf44a418 WatchSource:0}: Error finding container 3ff54e224ce5d8b032ae7003d514a61bb74fc004507815319ccc1fcbdf44a418: Status 404 returned error can't find the container with id 3ff54e224ce5d8b032ae7003d514a61bb74fc004507815319ccc1fcbdf44a418 Sep 11 18:49:54.668216 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:54.668175 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h928z" event={"ID":"bebc8327-abaa-4039-96f8-90928ef2e27d","Type":"ContainerStarted","Data":"3ff54e224ce5d8b032ae7003d514a61bb74fc004507815319ccc1fcbdf44a418"} Sep 11 18:49:54.670018 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:54.669981 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" event={"ID":"7c52ab3a-81f5-406d-b050-d9d35ea5ba38","Type":"ContainerStarted","Data":"e54d155f31412de991ba099acdca682fc7f228ea6fc505fe380ccdec2e4e96a6"} Sep 11 18:49:54.670149 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:54.670027 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" event={"ID":"7c52ab3a-81f5-406d-b050-d9d35ea5ba38","Type":"ContainerStarted","Data":"d6affd9d22a1c6b49a0be265399e1ccaec97727d1a711eeee507e3313738799f"} Sep 11 18:49:54.670149 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:54.670107 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:49:54.703476 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:54.703419 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" podStartSLOduration=99.703403592 podStartE2EDuration="1m39.703403592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:54.702765308 +0000 UTC m=+100.156571263" watchObservedRunningTime="2026-09-11 18:49:54.703403592 +0000 UTC m=+100.157209547" Sep 11 18:49:55.346528 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:55.346495 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc"] Sep 11 18:49:55.349968 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:55.349944 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc" Sep 11 18:49:55.354610 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:55.354570 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"open-cluster-management-image-pull-credentials\" is forbidden: User \"system:node:ip-10-0-143-253.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-143-253.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret" Sep 11 18:49:55.355156 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:55.355113 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-143-253.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-143-253.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 11 18:49:55.355156 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:55.355132 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"work-manager-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-143-253.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-143-253.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" type="*v1.Secret" Sep 11 18:49:55.355313 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:49:55.355177 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-143-253.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-143-253.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 11 18:49:55.365025 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:55.365003 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc"] Sep 11 18:49:55.411081 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:55.411055 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgw6k\" (UniqueName: \"kubernetes.io/projected/e94054d7-a249-4bba-be3a-8e42c97dad12-kube-api-access-kgw6k\") pod \"klusterlet-addon-workmgr-6d9bc4867d-x5tmc\" (UID: \"e94054d7-a249-4bba-be3a-8e42c97dad12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc" Sep 11 18:49:55.411197 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:55.411092 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e94054d7-a249-4bba-be3a-8e42c97dad12-tmp\") pod \"klusterlet-addon-workmgr-6d9bc4867d-x5tmc\" (UID: \"e94054d7-a249-4bba-be3a-8e42c97dad12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc" Sep 11 18:49:55.411197 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:55.411133 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e94054d7-a249-4bba-be3a-8e42c97dad12-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d9bc4867d-x5tmc\" (UID: \"e94054d7-a249-4bba-be3a-8e42c97dad12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc" Sep 11 18:49:55.508597 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:55.508523 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:55.512474 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:55.512448 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e94054d7-a249-4bba-be3a-8e42c97dad12-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d9bc4867d-x5tmc\" (UID: \"e94054d7-a249-4bba-be3a-8e42c97dad12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc" Sep 11 18:49:55.512596 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:55.512523 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgw6k\" (UniqueName: \"kubernetes.io/projected/e94054d7-a249-4bba-be3a-8e42c97dad12-kube-api-access-kgw6k\") pod \"klusterlet-addon-workmgr-6d9bc4867d-x5tmc\" (UID: \"e94054d7-a249-4bba-be3a-8e42c97dad12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc" Sep 11 18:49:55.512596 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:55.512566 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e94054d7-a249-4bba-be3a-8e42c97dad12-tmp\") pod \"klusterlet-addon-workmgr-6d9bc4867d-x5tmc\" (UID: \"e94054d7-a249-4bba-be3a-8e42c97dad12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc" Sep 11 18:49:55.512921 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:55.512904 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e94054d7-a249-4bba-be3a-8e42c97dad12-tmp\") pod \"klusterlet-addon-workmgr-6d9bc4867d-x5tmc\" (UID: \"e94054d7-a249-4bba-be3a-8e42c97dad12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc" Sep 11 18:49:55.638522 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:55.638494 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:55.676162 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:55.676126 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h928z" event={"ID":"bebc8327-abaa-4039-96f8-90928ef2e27d","Type":"ContainerStarted","Data":"fee975214da55e66f56b6b1a41b821298806f0d7004419e50d9b6cfb0ea30d3c"} Sep 11 18:49:55.806833 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:55.806805 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:55.943004 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:55.942929 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:56.264517 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:56.264428 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 11 18:49:56.276067 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:56.276035 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e94054d7-a249-4bba-be3a-8e42c97dad12-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d9bc4867d-x5tmc\" (UID: \"e94054d7-a249-4bba-be3a-8e42c97dad12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc" Sep 11 18:49:56.345573 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:56.345531 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 18:49:56.390045 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:56.390022 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 18:49:56.680170 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:56.680140 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h928z" event={"ID":"bebc8327-abaa-4039-96f8-90928ef2e27d","Type":"ContainerStarted","Data":"e55c4b8134d7f6531089009f08691a4853e40a4f963cca11fe52d0e1f5465099"} Sep 11 18:49:56.680313 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:56.680260 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h928z" Sep 11 18:49:56.688191 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:56.688174 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 18:49:56.696294 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:56.696268 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgw6k\" (UniqueName: \"kubernetes.io/projected/e94054d7-a249-4bba-be3a-8e42c97dad12-kube-api-access-kgw6k\") pod \"klusterlet-addon-workmgr-6d9bc4867d-x5tmc\" (UID: \"e94054d7-a249-4bba-be3a-8e42c97dad12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc" Sep 11 18:49:56.708230 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:56.708191 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-h928z" podStartSLOduration=66.266802296 podStartE2EDuration="1m7.708177136s" totalImagesPullingTime="1.44137484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:56.707639204 +0000 UTC m=+102.161445170" watchObservedRunningTime="2026-09-11 18:49:56.708177136 +0000 UTC m=+102.161983090" Sep 11 18:49:56.861115 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:56.861084 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc" Sep 11 18:49:57.132971 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:57.132943 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc"] Sep 11 18:49:57.135768 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:49:57.135741 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94054d7_a249_4bba_be3a_8e42c97dad12.slice/crio-5bea5cf4e226e8cdeeedbc1ebc6b4bd9e4dfa52f678d647652a396459c4d9cb6 WatchSource:0}: Error finding container 5bea5cf4e226e8cdeeedbc1ebc6b4bd9e4dfa52f678d647652a396459c4d9cb6: Status 404 returned error can't find the container with id 5bea5cf4e226e8cdeeedbc1ebc6b4bd9e4dfa52f678d647652a396459c4d9cb6 Sep 11 18:49:57.685130 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:57.685098 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc" event={"ID":"e94054d7-a249-4bba-be3a-8e42c97dad12","Type":"ContainerStarted","Data":"5bea5cf4e226e8cdeeedbc1ebc6b4bd9e4dfa52f678d647652a396459c4d9cb6"} Sep 11 18:49:58.891900 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:58.891860 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:58.989147 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:58.989074 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:59.105302 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:59.105257 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:59.175969 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:59.175810 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:49:59.573603 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:49:59.573570 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-grcgm" Sep 11 18:50:01.206648 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.206613 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:01.305654 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.305622 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:01.389697 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.389610 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:01.390301 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.390283 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rwl8w"] Sep 11 18:50:01.412653 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.412629 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rwl8w"] Sep 11 18:50:01.412786 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.412713 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rwl8w" Sep 11 18:50:01.415131 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.415111 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-255th\"" Sep 11 18:50:01.415262 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.415096 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 11 18:50:01.563237 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.563193 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/035c9e13-4e2a-41dc-b036-d50ec80a0cb6-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rwl8w\" (UID: \"035c9e13-4e2a-41dc-b036-d50ec80a0cb6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rwl8w" Sep 11 18:50:01.663704 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.663626 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/035c9e13-4e2a-41dc-b036-d50ec80a0cb6-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rwl8w\" (UID: \"035c9e13-4e2a-41dc-b036-d50ec80a0cb6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rwl8w" Sep 11 18:50:01.666034 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.666002 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/035c9e13-4e2a-41dc-b036-d50ec80a0cb6-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rwl8w\" (UID: \"035c9e13-4e2a-41dc-b036-d50ec80a0cb6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rwl8w" Sep 11 18:50:01.700714 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.700680 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc" event={"ID":"e94054d7-a249-4bba-be3a-8e42c97dad12","Type":"ContainerStarted","Data":"008f13a8e3e3477821e2b5abb972955a725eaa716c160cc728b2f1090b4dd410"} Sep 11 18:50:01.700874 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.700855 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc" Sep 11 18:50:01.702459 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.702439 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc" Sep 11 18:50:01.721700 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.721660 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d9bc4867d-x5tmc" podStartSLOduration=2.768309474 podStartE2EDuration="6.721646024s" totalImagesPullingTime="3.95333655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:01.721295425 +0000 UTC m=+107.175101376" watchObservedRunningTime="2026-09-11 18:50:01.721646024 +0000 UTC m=+107.175451978" Sep 11 18:50:01.726089 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.726070 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rwl8w" Sep 11 18:50:01.728762 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:01.728739 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:02.012891 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:02.012819 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rwl8w"] Sep 11 18:50:02.015517 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:50:02.015488 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod035c9e13_4e2a_41dc_b036_d50ec80a0cb6.slice/crio-b1e7fa9311e19827a5d9f9f21f9e320b54cb1b1201408c05c640f55720f57716 WatchSource:0}: Error finding container b1e7fa9311e19827a5d9f9f21f9e320b54cb1b1201408c05c640f55720f57716: Status 404 returned error can't find the container with id b1e7fa9311e19827a5d9f9f21f9e320b54cb1b1201408c05c640f55720f57716 Sep 11 18:50:02.706314 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:02.706276 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rwl8w" event={"ID":"035c9e13-4e2a-41dc-b036-d50ec80a0cb6","Type":"ContainerStarted","Data":"b1e7fa9311e19827a5d9f9f21f9e320b54cb1b1201408c05c640f55720f57716"} Sep 11 18:50:03.715981 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:03.715947 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:04.713745 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:04.713704 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rwl8w" event={"ID":"035c9e13-4e2a-41dc-b036-d50ec80a0cb6","Type":"ContainerStarted","Data":"b0df996963d534d59fd907bf81b8f3f3210e3652e8059782a10a23201d0adef8"} Sep 11 18:50:04.713930 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:04.713849 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rwl8w" Sep 11 18:50:04.718274 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:04.718251 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rwl8w" Sep 11 18:50:04.729792 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:04.729752 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rwl8w" podStartSLOduration=1.986753808 podStartE2EDuration="3.729738724s" totalImagesPullingTime="1.742984916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:04.729185805 +0000 UTC m=+110.182991772" watchObservedRunningTime="2026-09-11 18:50:04.729738724 +0000 UTC m=+110.183544678" Sep 11 18:50:05.415432 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:05.415399 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:05.487738 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:05.487698 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:05.607268 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:05.607240 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:06.687567 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:06.687526 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h928z" Sep 11 18:50:11.583816 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:11.583785 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:11.680392 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:11.680359 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:12.554249 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:12.554217 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:12.628713 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:12.628686 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:13.917469 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:13.917429 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:14.018284 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:14.018254 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:14.538140 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:14.538095 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:15.608166 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:15.608131 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:15.679942 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:15.679917 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:50:15.706117 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:15.706090 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:15.888481 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:15.888393 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:15.963138 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:15.963104 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:16.301775 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:16.301734 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:16.406215 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:16.406183 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:16.768502 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:16.768414 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:16.944068 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:16.944029 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:17.119739 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.119701 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:17.178901 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.178875 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r"] Sep 11 18:50:17.181479 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.181454 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.184762 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.184743 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 11 18:50:17.186014 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.185994 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vqk6s\"" Sep 11 18:50:17.186290 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.186273 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 11 18:50:17.187732 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.187430 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 18:50:17.187732 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.187712 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 18:50:17.188228 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.187952 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 18:50:17.190310 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.188689 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 11 18:50:17.195838 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.195814 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l6cw9"] Sep 11 18:50:17.198205 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.198185 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.201343 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.201324 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2wfkj\"" Sep 11 18:50:17.202089 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.202067 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 18:50:17.202197 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.202179 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 18:50:17.202259 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.202228 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 18:50:17.205317 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.205212 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r"] Sep 11 18:50:17.278704 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.278675 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c357c65-fe42-4459-b6e3-f80fbe5ac120-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.278859 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.278720 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6c357c65-fe42-4459-b6e3-f80fbe5ac120-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.278859 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.278753 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6c357c65-fe42-4459-b6e3-f80fbe5ac120-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.278859 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.278803 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c357c65-fe42-4459-b6e3-f80fbe5ac120-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.278859 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.278856 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dqvx\" (UniqueName: \"kubernetes.io/projected/6c357c65-fe42-4459-b6e3-f80fbe5ac120-kube-api-access-4dqvx\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.279068 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.278958 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c357c65-fe42-4459-b6e3-f80fbe5ac120-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.279832 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.279793 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:17.379860 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.379767 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-node-exporter-textfile\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.379860 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.379816 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c357c65-fe42-4459-b6e3-f80fbe5ac120-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.379860 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.379843 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c357c65-fe42-4459-b6e3-f80fbe5ac120-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.380112 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.379874 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.380112 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.379906 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6c357c65-fe42-4459-b6e3-f80fbe5ac120-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.380112 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.379929 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-node-exporter-accelerators-collector-config\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.380112 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.379957 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6c357c65-fe42-4459-b6e3-f80fbe5ac120-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.380112 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.379986 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c357c65-fe42-4459-b6e3-f80fbe5ac120-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.380112 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.380009 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-metrics-client-ca\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.380112 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.380051 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwwl4\" (UniqueName: \"kubernetes.io/projected/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-kube-api-access-pwwl4\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.380112 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.380097 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-node-exporter-tls\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.380483 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.380119 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-node-exporter-wtmp\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.380483 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.380154 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dqvx\" (UniqueName: \"kubernetes.io/projected/6c357c65-fe42-4459-b6e3-f80fbe5ac120-kube-api-access-4dqvx\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.380483 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.380197 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-sys\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.380483 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.380235 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-root\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.380483 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.380347 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6c357c65-fe42-4459-b6e3-f80fbe5ac120-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.380710 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.380590 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c357c65-fe42-4459-b6e3-f80fbe5ac120-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.380710 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.380597 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6c357c65-fe42-4459-b6e3-f80fbe5ac120-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.382458 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.382437 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c357c65-fe42-4459-b6e3-f80fbe5ac120-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.382521 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.382501 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c357c65-fe42-4459-b6e3-f80fbe5ac120-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.390674 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.390648 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dqvx\" (UniqueName: \"kubernetes.io/projected/6c357c65-fe42-4459-b6e3-f80fbe5ac120-kube-api-access-4dqvx\") pod \"kube-state-metrics-59fbdd77d-xqr2r\" (UID: \"6c357c65-fe42-4459-b6e3-f80fbe5ac120\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.403375 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.403352 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:17.480708 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.480680 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-node-exporter-tls\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.480708 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.480708 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-node-exporter-wtmp\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.480893 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.480734 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-sys\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.480893 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.480779 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-sys\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.480893 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.480847 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-root\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.480893 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.480854 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-node-exporter-wtmp\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.480893 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.480878 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-root\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.481135 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.480912 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-node-exporter-textfile\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.481135 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.480930 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.481135 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.480956 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-node-exporter-accelerators-collector-config\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.481135 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.480979 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-metrics-client-ca\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.481135 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.480999 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwwl4\" (UniqueName: \"kubernetes.io/projected/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-kube-api-access-pwwl4\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.481302 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.481252 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-node-exporter-textfile\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.481487 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.481466 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-node-exporter-accelerators-collector-config\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.481594 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.481573 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-metrics-client-ca\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.483078 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.483057 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-node-exporter-tls\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.483148 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.483112 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.490457 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.490429 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6468cc4b89-rwvgm"] Sep 11 18:50:17.497400 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.497382 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" Sep 11 18:50:17.497972 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.497956 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwwl4\" (UniqueName: \"kubernetes.io/projected/a5be5812-3f19-4d44-adf4-3c5de1c36cf1-kube-api-access-pwwl4\") pod \"node-exporter-l6cw9\" (UID: \"a5be5812-3f19-4d44-adf4-3c5de1c36cf1\") " pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.510037 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.510016 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l6cw9" Sep 11 18:50:17.519094 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:50:17.519067 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5be5812_3f19_4d44_adf4_3c5de1c36cf1.slice/crio-5bad0978431b83800d60017c9d137039fc15998ecb663280d1455dafc8b0e9db WatchSource:0}: Error finding container 5bad0978431b83800d60017c9d137039fc15998ecb663280d1455dafc8b0e9db: Status 404 returned error can't find the container with id 5bad0978431b83800d60017c9d137039fc15998ecb663280d1455dafc8b0e9db Sep 11 18:50:17.519759 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.519732 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:17.614466 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.614431 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:17.713888 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.713785 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:17.757055 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.757027 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l6cw9" event={"ID":"a5be5812-3f19-4d44-adf4-3c5de1c36cf1","Type":"ContainerStarted","Data":"5bad0978431b83800d60017c9d137039fc15998ecb663280d1455dafc8b0e9db"} Sep 11 18:50:17.774872 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.774852 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r"] Sep 11 18:50:17.776939 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:50:17.776917 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c357c65_fe42_4459_b6e3_f80fbe5ac120.slice/crio-b3d437d2dd0edb2079b6e320f091b4e5840cf2d810938dcc876cc2f0b2c307e1 WatchSource:0}: Error finding container b3d437d2dd0edb2079b6e320f091b4e5840cf2d810938dcc876cc2f0b2c307e1: Status 404 returned error can't find the container with id b3d437d2dd0edb2079b6e320f091b4e5840cf2d810938dcc876cc2f0b2c307e1 Sep 11 18:50:17.818214 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.818184 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:17.919616 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:17.919583 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:18.053276 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:18.053242 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:18.167051 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:18.167022 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:18.292426 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:18.292340 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:18.761683 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:18.761642 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" event={"ID":"6c357c65-fe42-4459-b6e3-f80fbe5ac120","Type":"ContainerStarted","Data":"b3d437d2dd0edb2079b6e320f091b4e5840cf2d810938dcc876cc2f0b2c307e1"} Sep 11 18:50:18.763460 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:18.763424 2556 generic.go:356] "Generic (PLEG): container finished" podID="a5be5812-3f19-4d44-adf4-3c5de1c36cf1" containerID="80fc139d16d4dd519d929f3b9a84f38a7df6f5e752e5d58717ca9903ae82ae30" exitCode=0 Sep 11 18:50:18.763607 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:18.763485 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l6cw9" event={"ID":"a5be5812-3f19-4d44-adf4-3c5de1c36cf1","Type":"ContainerDied","Data":"80fc139d16d4dd519d929f3b9a84f38a7df6f5e752e5d58717ca9903ae82ae30"} Sep 11 18:50:19.769077 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:19.769040 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" event={"ID":"6c357c65-fe42-4459-b6e3-f80fbe5ac120","Type":"ContainerStarted","Data":"36864726ff27d77329a1b6ca1436adfab5661257336450d2ca5d343e29709224"} Sep 11 18:50:19.769510 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:19.769100 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" event={"ID":"6c357c65-fe42-4459-b6e3-f80fbe5ac120","Type":"ContainerStarted","Data":"4cf1881827591891235e9915fbc6e6a7035e7714f0f19495253e536b4efbdef3"} Sep 11 18:50:19.769510 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:19.769134 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" event={"ID":"6c357c65-fe42-4459-b6e3-f80fbe5ac120","Type":"ContainerStarted","Data":"81551e4fb49c3f42383815f82f4ea1915ab8622eb95f1817525f2f5c1d614038"} Sep 11 18:50:19.771104 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:19.771081 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l6cw9" event={"ID":"a5be5812-3f19-4d44-adf4-3c5de1c36cf1","Type":"ContainerStarted","Data":"9a503cafd7b6a2c6522d86fc5812b329a8f37134cfb53de53ca571a1127cd68b"} Sep 11 18:50:19.771192 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:19.771126 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l6cw9" event={"ID":"a5be5812-3f19-4d44-adf4-3c5de1c36cf1","Type":"ContainerStarted","Data":"39a89651d727a4a8b263ec5110ed44300ba85bb70ea855a530526dd0287af924"} Sep 11 18:50:19.792609 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:19.792569 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-xqr2r" podStartSLOduration=1.5083912929999999 podStartE2EDuration="2.792554486s" totalImagesPullingTime="1.284163193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:19.79150574 +0000 UTC m=+125.245311710" watchObservedRunningTime="2026-09-11 18:50:19.792554486 +0000 UTC m=+125.246360432" Sep 11 18:50:19.814425 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:19.814383 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l6cw9" podStartSLOduration=1.72255095 podStartE2EDuration="2.814370965s" totalImagesPullingTime="1.022806805s" totalInitContainerRuntime="69.01321ms" isStatefulPod=true podCreationTimestamp="2026-09-11 18:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:19.813463805 +0000 UTC m=+125.267269763" watchObservedRunningTime="2026-09-11 18:50:19.814370965 +0000 UTC m=+125.268176919" Sep 11 18:50:19.961996 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:19.961953 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:20.116724 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.116696 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b95dc5876-cchsf"] Sep 11 18:50:20.120118 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.120092 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.122214 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.122192 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 11 18:50:20.123035 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.123017 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 11 18:50:20.123149 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.123073 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 11 18:50:20.123149 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.123020 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 11 18:50:20.123888 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.123867 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 11 18:50:20.123994 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.123874 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a003goqi9l778\"" Sep 11 18:50:20.123994 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.123975 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zvt6r\"" Sep 11 18:50:20.132520 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.132493 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:20.148764 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.148740 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b95dc5876-cchsf"] Sep 11 18:50:20.281066 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.281024 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:20.308165 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.308141 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-grpc-tls\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.308299 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.308170 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gkkw\" (UniqueName: \"kubernetes.io/projected/d9743bf3-35ce-43c1-8fcd-331b9f01538c-kube-api-access-5gkkw\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.308299 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.308192 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-thanos-querier-tls\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.308299 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.308283 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.308406 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.308315 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.308406 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.308340 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.308406 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.308397 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.308518 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.308416 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9743bf3-35ce-43c1-8fcd-331b9f01538c-metrics-client-ca\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.409264 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.409184 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-grpc-tls\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.409264 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.409222 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gkkw\" (UniqueName: \"kubernetes.io/projected/d9743bf3-35ce-43c1-8fcd-331b9f01538c-kube-api-access-5gkkw\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.409264 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.409252 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-thanos-querier-tls\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.409501 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.409321 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.409501 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.409459 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.409630 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.409526 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.409630 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.409618 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.409779 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.409648 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9743bf3-35ce-43c1-8fcd-331b9f01538c-metrics-client-ca\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.410407 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.410378 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9743bf3-35ce-43c1-8fcd-331b9f01538c-metrics-client-ca\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.412016 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.411992 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.412166 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.412141 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.412364 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.412344 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.412429 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.412361 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-thanos-querier-tls\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.412429 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.412377 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.412582 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.412562 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9743bf3-35ce-43c1-8fcd-331b9f01538c-secret-grpc-tls\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.421759 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.421739 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gkkw\" (UniqueName: \"kubernetes.io/projected/d9743bf3-35ce-43c1-8fcd-331b9f01538c-kube-api-access-5gkkw\") pod \"thanos-querier-b95dc5876-cchsf\" (UID: \"d9743bf3-35ce-43c1-8fcd-331b9f01538c\") " pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.430020 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.430005 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:20.715079 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.714991 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b95dc5876-cchsf"] Sep 11 18:50:20.718953 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:50:20.718925 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9743bf3_35ce_43c1_8fcd_331b9f01538c.slice/crio-1734b945a8a51269c22fed5b0e85b9a64f2e914a280c5b5c1e9abcd38a269551 WatchSource:0}: Error finding container 1734b945a8a51269c22fed5b0e85b9a64f2e914a280c5b5c1e9abcd38a269551: Status 404 returned error can't find the container with id 1734b945a8a51269c22fed5b0e85b9a64f2e914a280c5b5c1e9abcd38a269551 Sep 11 18:50:20.775421 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:20.775381 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" event={"ID":"d9743bf3-35ce-43c1-8fcd-331b9f01538c","Type":"ContainerStarted","Data":"1734b945a8a51269c22fed5b0e85b9a64f2e914a280c5b5c1e9abcd38a269551"} Sep 11 18:50:21.548555 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.548470 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:21.656862 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.656829 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-hqr5j"] Sep 11 18:50:21.660909 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.660883 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hqr5j" Sep 11 18:50:21.663977 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.663957 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 11 18:50:21.667845 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.667823 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jr5qn\"" Sep 11 18:50:21.687968 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.687893 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-hqr5j"] Sep 11 18:50:21.720257 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.720226 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ce6c5f05-f277-4907-a227-8296e0ab8f4a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-hqr5j\" (UID: \"ce6c5f05-f277-4907-a227-8296e0ab8f4a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hqr5j" Sep 11 18:50:21.720441 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.720293 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25hkk\" (UniqueName: \"kubernetes.io/projected/ce6c5f05-f277-4907-a227-8296e0ab8f4a-kube-api-access-25hkk\") pod \"monitoring-plugin-5d85778b9d-hqr5j\" (UID: \"ce6c5f05-f277-4907-a227-8296e0ab8f4a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hqr5j" Sep 11 18:50:21.721345 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.721309 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:21.778106 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.778071 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-57dc7f4fdd-72fr6"] Sep 11 18:50:21.782213 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.782187 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.786264 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.786239 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 11 18:50:21.786392 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.786262 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 18:50:21.786440 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.786415 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-caiiblv2rv15s\"" Sep 11 18:50:21.786505 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.786487 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 11 18:50:21.786614 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.786593 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 11 18:50:21.786988 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.786963 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bdszw\"" Sep 11 18:50:21.794594 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.794574 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57dc7f4fdd-72fr6"] Sep 11 18:50:21.820898 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.820833 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/639dfeb3-c288-4575-89a8-80647f6437d6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.820898 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.820872 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/639dfeb3-c288-4575-89a8-80647f6437d6-metrics-server-audit-profiles\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.821057 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.820901 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/639dfeb3-c288-4575-89a8-80647f6437d6-client-ca-bundle\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.821057 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.820957 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnfds\" (UniqueName: \"kubernetes.io/projected/639dfeb3-c288-4575-89a8-80647f6437d6-kube-api-access-cnfds\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.821057 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.820998 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/639dfeb3-c288-4575-89a8-80647f6437d6-audit-log\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.821057 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.821046 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ce6c5f05-f277-4907-a227-8296e0ab8f4a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-hqr5j\" (UID: \"ce6c5f05-f277-4907-a227-8296e0ab8f4a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hqr5j" Sep 11 18:50:21.821256 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.821231 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/639dfeb3-c288-4575-89a8-80647f6437d6-secret-metrics-server-tls\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.821334 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.821276 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25hkk\" (UniqueName: \"kubernetes.io/projected/ce6c5f05-f277-4907-a227-8296e0ab8f4a-kube-api-access-25hkk\") pod \"monitoring-plugin-5d85778b9d-hqr5j\" (UID: \"ce6c5f05-f277-4907-a227-8296e0ab8f4a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hqr5j" Sep 11 18:50:21.821334 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.821324 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/639dfeb3-c288-4575-89a8-80647f6437d6-secret-metrics-server-client-certs\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.823779 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.823757 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ce6c5f05-f277-4907-a227-8296e0ab8f4a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-hqr5j\" (UID: \"ce6c5f05-f277-4907-a227-8296e0ab8f4a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hqr5j" Sep 11 18:50:21.824073 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.823992 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:21.829481 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.829459 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25hkk\" (UniqueName: \"kubernetes.io/projected/ce6c5f05-f277-4907-a227-8296e0ab8f4a-kube-api-access-25hkk\") pod \"monitoring-plugin-5d85778b9d-hqr5j\" (UID: \"ce6c5f05-f277-4907-a227-8296e0ab8f4a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hqr5j" Sep 11 18:50:21.920737 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.920692 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:21.922114 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.922084 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/639dfeb3-c288-4575-89a8-80647f6437d6-secret-metrics-server-tls\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.922224 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.922121 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/639dfeb3-c288-4575-89a8-80647f6437d6-secret-metrics-server-client-certs\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.922224 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.922161 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/639dfeb3-c288-4575-89a8-80647f6437d6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.922224 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.922180 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/639dfeb3-c288-4575-89a8-80647f6437d6-metrics-server-audit-profiles\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.922224 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.922194 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/639dfeb3-c288-4575-89a8-80647f6437d6-client-ca-bundle\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.922224 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.922214 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnfds\" (UniqueName: \"kubernetes.io/projected/639dfeb3-c288-4575-89a8-80647f6437d6-kube-api-access-cnfds\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.922451 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.922247 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/639dfeb3-c288-4575-89a8-80647f6437d6-audit-log\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.922702 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.922676 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/639dfeb3-c288-4575-89a8-80647f6437d6-audit-log\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.922993 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.922971 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/639dfeb3-c288-4575-89a8-80647f6437d6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.923156 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.923137 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/639dfeb3-c288-4575-89a8-80647f6437d6-metrics-server-audit-profiles\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.924663 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.924645 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/639dfeb3-c288-4575-89a8-80647f6437d6-client-ca-bundle\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.925017 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.924999 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/639dfeb3-c288-4575-89a8-80647f6437d6-secret-metrics-server-tls\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.925132 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.925103 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/639dfeb3-c288-4575-89a8-80647f6437d6-secret-metrics-server-client-certs\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.945756 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.945737 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnfds\" (UniqueName: \"kubernetes.io/projected/639dfeb3-c288-4575-89a8-80647f6437d6-kube-api-access-cnfds\") pod \"metrics-server-57dc7f4fdd-72fr6\" (UID: \"639dfeb3-c288-4575-89a8-80647f6437d6\") " pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:21.972609 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:21.972589 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hqr5j" Sep 11 18:50:22.092562 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:22.092459 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:22.253671 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:22.253635 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-hqr5j"] Sep 11 18:50:22.565318 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:50:22.565280 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6c5f05_f277_4907_a227_8296e0ab8f4a.slice/crio-d2f23d5a9513e7babdd3ebe915e59c95c5d3c29c16e9443bc398196540d50391 WatchSource:0}: Error finding container d2f23d5a9513e7babdd3ebe915e59c95c5d3c29c16e9443bc398196540d50391: Status 404 returned error can't find the container with id d2f23d5a9513e7babdd3ebe915e59c95c5d3c29c16e9443bc398196540d50391 Sep 11 18:50:22.783830 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:22.783801 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hqr5j" event={"ID":"ce6c5f05-f277-4907-a227-8296e0ab8f4a","Type":"ContainerStarted","Data":"d2f23d5a9513e7babdd3ebe915e59c95c5d3c29c16e9443bc398196540d50391"} Sep 11 18:50:22.785376 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:22.785355 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" event={"ID":"d9743bf3-35ce-43c1-8fcd-331b9f01538c","Type":"ContainerStarted","Data":"0eb07301e27c5253eec38a4d557aad9ec5bbb24e6f0876157594d87f2c486adb"} Sep 11 18:50:22.785463 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:22.785407 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" event={"ID":"d9743bf3-35ce-43c1-8fcd-331b9f01538c","Type":"ContainerStarted","Data":"de41c52325be1662e804d2418e315261bd9cc151f75ff6e0858be40e4d943995"} Sep 11 18:50:22.845457 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:22.845435 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57dc7f4fdd-72fr6"] Sep 11 18:50:22.847703 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:50:22.847678 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639dfeb3_c288_4575_89a8_80647f6437d6.slice/crio-01c71535ab80681cf73fb142eec6f5fda633a13740ad2c3b80f3368676c4b57f WatchSource:0}: Error finding container 01c71535ab80681cf73fb142eec6f5fda633a13740ad2c3b80f3368676c4b57f: Status 404 returned error can't find the container with id 01c71535ab80681cf73fb142eec6f5fda633a13740ad2c3b80f3368676c4b57f Sep 11 18:50:23.372699 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.372665 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 18:50:23.377366 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.376931 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.377366 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.377028 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:23.381515 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.381491 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zcqtd\"" Sep 11 18:50:23.381772 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.381744 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 11 18:50:23.381772 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.381755 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5vt1hohp9c68n\"" Sep 11 18:50:23.382466 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.382418 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 11 18:50:23.383678 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.383493 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 11 18:50:23.383678 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.383668 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 11 18:50:23.384126 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.383838 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 11 18:50:23.384126 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.383939 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 11 18:50:23.384126 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.384062 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 11 18:50:23.384126 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.384076 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 11 18:50:23.384126 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.384094 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 11 18:50:23.386275 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.384949 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 11 18:50:23.386275 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.385154 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 11 18:50:23.386275 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.385457 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 11 18:50:23.386275 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.386216 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 11 18:50:23.391853 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.391834 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 11 18:50:23.403458 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.403433 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 18:50:23.438704 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.438673 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-config-out\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.438831 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.438719 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.438831 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.438777 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.438831 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.438808 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-config\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.438999 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.438832 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.438999 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.438908 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.438999 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.438945 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.438999 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.438967 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.438999 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.438989 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.439231 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.439043 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.439231 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.439074 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.439231 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.439106 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.439231 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.439127 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.439231 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.439147 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.439231 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.439173 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-web-config\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.439231 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.439212 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.439589 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.439247 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.439589 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.439278 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.439589 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.439303 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.439589 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.439320 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmrrq\" (UniqueName: \"kubernetes.io/projected/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-kube-api-access-xmrrq\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.507508 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.507433 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.540610 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.540661 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.540687 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrrq\" (UniqueName: \"kubernetes.io/projected/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-kube-api-access-xmrrq\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.540739 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-config-out\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.540767 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.540824 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.540853 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-config\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.540877 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.540905 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.540927 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.540952 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.540978 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.541008 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.541040 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.541065 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.541089 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.541712 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.541113 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.542664 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.541140 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-web-config\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.542664 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.541187 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.542664 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.541234 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.543285 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.543254 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.544577 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.544534 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.549224 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.545699 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.549224 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.545777 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.549224 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.548122 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.549224 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.548247 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.549224 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.548775 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.549691 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.549630 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.552376 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.552354 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.555000 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.554980 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-web-config\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.555153 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.554980 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.556304 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.556276 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-config-out\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.556397 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.556374 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.556458 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.556421 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.556458 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.556451 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-config\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.557066 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.557041 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.557386 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.557363 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.557750 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.557709 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmrrq\" (UniqueName: \"kubernetes.io/projected/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-kube-api-access-xmrrq\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.558794 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.558773 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.560763 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.560721 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.689996 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.689679 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:23.708038 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.707988 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:23.773560 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.773503 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:23.799065 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.799026 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" event={"ID":"d9743bf3-35ce-43c1-8fcd-331b9f01538c","Type":"ContainerStarted","Data":"16ae0abd0a80228ce81d47110bdb2a186aed549eb2f063da560344775a28d7af"} Sep 11 18:50:23.800655 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:23.800623 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" event={"ID":"639dfeb3-c288-4575-89a8-80647f6437d6","Type":"ContainerStarted","Data":"01c71535ab80681cf73fb142eec6f5fda633a13740ad2c3b80f3368676c4b57f"} Sep 11 18:50:24.033557 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:24.033505 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 18:50:24.274207 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:50:24.274179 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aec0263_5fdd_4cfc_a6cf_fa07dbcaca9f.slice/crio-b3f411960d1ff964c611897dea4eee790301286d667fdd534e97f49c67b72643 WatchSource:0}: Error finding container b3f411960d1ff964c611897dea4eee790301286d667fdd534e97f49c67b72643: Status 404 returned error can't find the container with id b3f411960d1ff964c611897dea4eee790301286d667fdd534e97f49c67b72643 Sep 11 18:50:24.807122 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:24.807090 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f","Type":"ContainerStarted","Data":"b3f411960d1ff964c611897dea4eee790301286d667fdd534e97f49c67b72643"} Sep 11 18:50:24.809423 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:24.809395 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hqr5j" event={"ID":"ce6c5f05-f277-4907-a227-8296e0ab8f4a","Type":"ContainerStarted","Data":"02fa5086a1c41128ca68b964fcaf363861fc3d9171dc8047b005803741effb0a"} Sep 11 18:50:24.810105 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:24.809731 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hqr5j" Sep 11 18:50:24.814578 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:24.814481 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" event={"ID":"d9743bf3-35ce-43c1-8fcd-331b9f01538c","Type":"ContainerStarted","Data":"a86c4d3bbec8ccf9306d919f1b66b0e6e86396b8918ed28fb5ef1af4d3958d77"} Sep 11 18:50:24.814681 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:24.814577 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" event={"ID":"d9743bf3-35ce-43c1-8fcd-331b9f01538c","Type":"ContainerStarted","Data":"d12730d31c9fb3da27a9e3272f8185cbd8af380c1cde04498b84f7a4e2d10b3b"} Sep 11 18:50:24.814681 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:24.814628 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:24.814681 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:24.814668 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" event={"ID":"d9743bf3-35ce-43c1-8fcd-331b9f01538c","Type":"ContainerStarted","Data":"a8b1febf23bb728f10b3c212937d0abf3bb150e7279d206fcbc227bdbeb5bcae"} Sep 11 18:50:24.816031 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:24.816008 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hqr5j" Sep 11 18:50:24.816399 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:24.816357 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" event={"ID":"639dfeb3-c288-4575-89a8-80647f6437d6","Type":"ContainerStarted","Data":"5771643d0682746ed8318492de4f9c63621acec24a95208edf695a7dae491c7c"} Sep 11 18:50:24.827243 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:24.827180 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-hqr5j" podStartSLOduration=1.743499283 podStartE2EDuration="3.827164912s" totalImagesPullingTime="2.083665629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:24.826413703 +0000 UTC m=+130.280219685" watchObservedRunningTime="2026-09-11 18:50:24.827164912 +0000 UTC m=+130.280970865" Sep 11 18:50:24.853616 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:24.853459 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs\") pod \"network-metrics-daemon-rhmpv\" (UID: \"681500e4-89ea-42a9-9d91-0d44e7698747\") " pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:50:24.856135 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:24.856109 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681500e4-89ea-42a9-9d91-0d44e7698747-metrics-certs\") pod \"network-metrics-daemon-rhmpv\" (UID: \"681500e4-89ea-42a9-9d91-0d44e7698747\") " pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:50:24.872948 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:24.872882 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" podStartSLOduration=2.072277366 podStartE2EDuration="3.872862798s" totalImagesPullingTime="1.800585432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:24.872215864 +0000 UTC m=+130.326021821" watchObservedRunningTime="2026-09-11 18:50:24.872862798 +0000 UTC m=+130.326668754" Sep 11 18:50:24.898376 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:24.898321 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" podStartSLOduration=2.037013906 podStartE2EDuration="4.898302135s" totalImagesPullingTime="2.861288229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:50:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:24.897106027 +0000 UTC m=+130.350912011" watchObservedRunningTime="2026-09-11 18:50:24.898302135 +0000 UTC m=+130.352108089" Sep 11 18:50:25.087524 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:25.087490 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kzbqc\"" Sep 11 18:50:25.095323 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:25.095297 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rhmpv" Sep 11 18:50:25.452120 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:25.452045 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rhmpv"] Sep 11 18:50:25.458691 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:50:25.458657 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681500e4_89ea_42a9_9d91_0d44e7698747.slice/crio-362aeab4af1d4980a9b411446fcb321546e017199527b9b3c58e62a01956854b WatchSource:0}: Error finding container 362aeab4af1d4980a9b411446fcb321546e017199527b9b3c58e62a01956854b: Status 404 returned error can't find the container with id 362aeab4af1d4980a9b411446fcb321546e017199527b9b3c58e62a01956854b Sep 11 18:50:25.821178 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:25.821138 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rhmpv" event={"ID":"681500e4-89ea-42a9-9d91-0d44e7698747","Type":"ContainerStarted","Data":"362aeab4af1d4980a9b411446fcb321546e017199527b9b3c58e62a01956854b"} Sep 11 18:50:25.822587 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:25.822564 2556 generic.go:356] "Generic (PLEG): container finished" podID="3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f" containerID="08c02595a83743e4c21cd8b028d96be262d74fc58acca522a2aa2dfca88ae05f" exitCode=0 Sep 11 18:50:25.822698 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:25.822625 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f","Type":"ContainerDied","Data":"08c02595a83743e4c21cd8b028d96be262d74fc58acca522a2aa2dfca88ae05f"} Sep 11 18:50:26.222177 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:26.222102 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:26.321845 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:26.321807 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:26.829890 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:26.829845 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rhmpv" event={"ID":"681500e4-89ea-42a9-9d91-0d44e7698747","Type":"ContainerStarted","Data":"4c93ff52fd4625bea55fd292439d18f667ea128229e6cadbb4d4eab92707064d"} Sep 11 18:50:26.830337 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:26.829914 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rhmpv" event={"ID":"681500e4-89ea-42a9-9d91-0d44e7698747","Type":"ContainerStarted","Data":"b7aaccf82a6be4b5d2c7140d0539409cfc55398f5b509c6897c9992e5ca6e5bd"} Sep 11 18:50:26.854870 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:26.854803 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rhmpv" podStartSLOduration=130.891461904 podStartE2EDuration="2m11.854780591s" totalImagesPullingTime="963.318687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:26.853291115 +0000 UTC m=+132.307097089" watchObservedRunningTime="2026-09-11 18:50:26.854780591 +0000 UTC m=+132.308586550" Sep 11 18:50:26.998470 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:26.998383 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:27.100845 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:27.100768 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:29.845904 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:29.845825 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f","Type":"ContainerStarted","Data":"c84ea23716b86ec6b814dddbfa83fea71d2475106bfa442acdd966580d64f292"} Sep 11 18:50:29.845904 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:29.845892 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f","Type":"ContainerStarted","Data":"0159990434af014f7bf27536f31ddac86e1098d976f25ae36b9aa6a90a30d970"} Sep 11 18:50:29.846271 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:29.845918 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f","Type":"ContainerStarted","Data":"5b8f53056046a02eab4304f5398855e10d246723207833bae013efcec74512f7"} Sep 11 18:50:29.846271 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:29.845943 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f","Type":"ContainerStarted","Data":"e8adfd0208de7dd835bdb39e98f4f3c0994edfcfda8b99f99ce4d911b2741798"} Sep 11 18:50:29.846271 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:29.845980 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f","Type":"ContainerStarted","Data":"0816b04097568ea8c23f37d85219ae8443a07ea3cdb9e628284cc3d83da47357"} Sep 11 18:50:29.846271 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:29.846009 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f","Type":"ContainerStarted","Data":"cee8e928f83a0a3fda665db5e7a312bee910eba30c5d40be4c2a9dd53d57675a"} Sep 11 18:50:29.881048 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:29.880982 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.393252152 podStartE2EDuration="6.880960597s" totalImagesPullingTime="4.468328055s" totalInitContainerRuntime="19.38039ms" isStatefulPod=false podCreationTimestamp="2026-09-11 18:50:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:29.879685562 +0000 UTC m=+135.333491538" watchObservedRunningTime="2026-09-11 18:50:29.880960597 +0000 UTC m=+135.334766631" Sep 11 18:50:30.828741 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:30.828711 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b95dc5876-cchsf" Sep 11 18:50:32.832025 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:32.831991 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:32.957624 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:32.957594 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:33.690315 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:33.690287 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:50:37.943718 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:37.943681 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:38.041405 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:38.041364 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:38.142965 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:38.142930 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:38.884723 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:38.884689 2556 generic.go:356] "Generic (PLEG): container finished" podID="8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12" containerID="c38cd56bf4efb32c1c73a07811f54ac3c394c9324e2ce9f746f97c75e6e22419" exitCode=0 Sep 11 18:50:38.884852 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:38.884762 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll" event={"ID":"8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12","Type":"ContainerDied","Data":"c38cd56bf4efb32c1c73a07811f54ac3c394c9324e2ce9f746f97c75e6e22419"} Sep 11 18:50:38.885076 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:38.885063 2556 scope.go:117] "RemoveContainer" containerID="c38cd56bf4efb32c1c73a07811f54ac3c394c9324e2ce9f746f97c75e6e22419" Sep 11 18:50:39.890378 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:39.890342 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-57sll" event={"ID":"8a8dd4b3-8dfa-4b38-9d18-70f2187fdb12","Type":"ContainerStarted","Data":"6a6a643c983dd982219e516c46522de8744edf634f61f064c9ee19a15db0824a"} Sep 11 18:50:42.093436 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.093386 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:42.093875 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.093464 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:50:42.512501 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.512395 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" podUID="7c52ab3a-81f5-406d-b050-d9d35ea5ba38" containerName="registry" containerID="cri-o://e54d155f31412de991ba099acdca682fc7f228ea6fc505fe380ccdec2e4e96a6" gracePeriod=30 Sep 11 18:50:42.749698 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.749675 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:50:42.813499 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.813472 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-trusted-ca\") pod \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " Sep 11 18:50:42.813663 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.813507 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-image-registry-private-configuration\") pod \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " Sep 11 18:50:42.813663 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.813531 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-certificates\") pod \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " Sep 11 18:50:42.813663 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.813617 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-459b4\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-kube-api-access-459b4\") pod \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " Sep 11 18:50:42.813663 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.813647 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-bound-sa-token\") pod \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " Sep 11 18:50:42.813879 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.813671 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-ca-trust-extracted\") pod \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " Sep 11 18:50:42.813879 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.813704 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls\") pod \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " Sep 11 18:50:42.813879 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.813736 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-installation-pull-secrets\") pod \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\" (UID: \"7c52ab3a-81f5-406d-b050-d9d35ea5ba38\") " Sep 11 18:50:42.814067 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.814040 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7c52ab3a-81f5-406d-b050-d9d35ea5ba38" (UID: "7c52ab3a-81f5-406d-b050-d9d35ea5ba38"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:50:42.814134 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.814071 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7c52ab3a-81f5-406d-b050-d9d35ea5ba38" (UID: "7c52ab3a-81f5-406d-b050-d9d35ea5ba38"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:50:42.816233 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.816201 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7c52ab3a-81f5-406d-b050-d9d35ea5ba38" (UID: "7c52ab3a-81f5-406d-b050-d9d35ea5ba38"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:50:42.816405 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.816378 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7c52ab3a-81f5-406d-b050-d9d35ea5ba38" (UID: "7c52ab3a-81f5-406d-b050-d9d35ea5ba38"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:50:42.816524 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.816431 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-kube-api-access-459b4" (OuterVolumeSpecName: "kube-api-access-459b4") pod "7c52ab3a-81f5-406d-b050-d9d35ea5ba38" (UID: "7c52ab3a-81f5-406d-b050-d9d35ea5ba38"). InnerVolumeSpecName "kube-api-access-459b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:50:42.816524 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.816470 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7c52ab3a-81f5-406d-b050-d9d35ea5ba38" (UID: "7c52ab3a-81f5-406d-b050-d9d35ea5ba38"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:50:42.816524 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.816514 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7c52ab3a-81f5-406d-b050-d9d35ea5ba38" (UID: "7c52ab3a-81f5-406d-b050-d9d35ea5ba38"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:50:42.822129 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.822108 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7c52ab3a-81f5-406d-b050-d9d35ea5ba38" (UID: "7c52ab3a-81f5-406d-b050-d9d35ea5ba38"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:50:42.903444 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.903413 2556 generic.go:356] "Generic (PLEG): container finished" podID="7c52ab3a-81f5-406d-b050-d9d35ea5ba38" containerID="e54d155f31412de991ba099acdca682fc7f228ea6fc505fe380ccdec2e4e96a6" exitCode=0 Sep 11 18:50:42.903616 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.903454 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" event={"ID":"7c52ab3a-81f5-406d-b050-d9d35ea5ba38","Type":"ContainerDied","Data":"e54d155f31412de991ba099acdca682fc7f228ea6fc505fe380ccdec2e4e96a6"} Sep 11 18:50:42.903616 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.903457 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" Sep 11 18:50:42.903616 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.903490 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6468cc4b89-rwvgm" event={"ID":"7c52ab3a-81f5-406d-b050-d9d35ea5ba38","Type":"ContainerDied","Data":"d6affd9d22a1c6b49a0be265399e1ccaec97727d1a711eeee507e3313738799f"} Sep 11 18:50:42.903616 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.903510 2556 scope.go:117] "RemoveContainer" containerID="e54d155f31412de991ba099acdca682fc7f228ea6fc505fe380ccdec2e4e96a6" Sep 11 18:50:42.912299 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.912281 2556 scope.go:117] "RemoveContainer" containerID="e54d155f31412de991ba099acdca682fc7f228ea6fc505fe380ccdec2e4e96a6" Sep 11 18:50:42.912619 ip-10-0-143-253 kubenswrapper[2556]: E0911 18:50:42.912599 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e54d155f31412de991ba099acdca682fc7f228ea6fc505fe380ccdec2e4e96a6\": container with ID starting with e54d155f31412de991ba099acdca682fc7f228ea6fc505fe380ccdec2e4e96a6 not found: ID does not exist" containerID="e54d155f31412de991ba099acdca682fc7f228ea6fc505fe380ccdec2e4e96a6" Sep 11 18:50:42.912692 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.912627 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e54d155f31412de991ba099acdca682fc7f228ea6fc505fe380ccdec2e4e96a6"} err="failed to get container status \"e54d155f31412de991ba099acdca682fc7f228ea6fc505fe380ccdec2e4e96a6\": rpc error: code = NotFound desc = could not find container \"e54d155f31412de991ba099acdca682fc7f228ea6fc505fe380ccdec2e4e96a6\": container with ID starting with e54d155f31412de991ba099acdca682fc7f228ea6fc505fe380ccdec2e4e96a6 not found: ID does not exist" Sep 11 18:50:42.914622 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.914603 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-459b4\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-kube-api-access-459b4\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 18:50:42.914680 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.914624 2556 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-bound-sa-token\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 18:50:42.914680 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.914633 2556 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-ca-trust-extracted\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 18:50:42.914680 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.914642 2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-tls\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 18:50:42.914680 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.914651 2556 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-installation-pull-secrets\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 18:50:42.914680 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.914660 2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-trusted-ca\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 18:50:42.914680 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.914669 2556 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-image-registry-private-configuration\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 18:50:42.914680 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.914678 2556 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c52ab3a-81f5-406d-b050-d9d35ea5ba38-registry-certificates\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 18:50:42.923917 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.923893 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6468cc4b89-rwvgm"] Sep 11 18:50:42.928453 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:42.928434 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6468cc4b89-rwvgm"] Sep 11 18:50:43.170831 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:43.170761 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c52ab3a-81f5-406d-b050-d9d35ea5ba38" path="/var/lib/kubelet/pods/7c52ab3a-81f5-406d-b050-d9d35ea5ba38/volumes" Sep 11 18:50:43.908611 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:43.908575 2556 generic.go:356] "Generic (PLEG): container finished" podID="8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1" containerID="e1b1a766058e453e25771e5397ee0e8896562897ac5e3c2c31e0bb51d21dfa33" exitCode=0 Sep 11 18:50:43.908751 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:43.908645 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk" event={"ID":"8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1","Type":"ContainerDied","Data":"e1b1a766058e453e25771e5397ee0e8896562897ac5e3c2c31e0bb51d21dfa33"} Sep 11 18:50:43.909075 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:43.909053 2556 scope.go:117] "RemoveContainer" containerID="e1b1a766058e453e25771e5397ee0e8896562897ac5e3c2c31e0bb51d21dfa33" Sep 11 18:50:44.915243 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:44.915206 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-m2xrk" event={"ID":"8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1","Type":"ContainerStarted","Data":"d49e9e0efeac834f4dd69d43825d08b334e634ffca95d55bedc68d9ab79f4837"} Sep 11 18:50:44.916626 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:44.916601 2556 generic.go:356] "Generic (PLEG): container finished" podID="9462a6cd-3ef7-4962-aaea-01017510998d" containerID="fd64c1875ad870b871b16c1ce664e7155bc373a52b92f1240292e23b4f1e7077" exitCode=0 Sep 11 18:50:44.916725 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:44.916682 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-kxkhf" event={"ID":"9462a6cd-3ef7-4962-aaea-01017510998d","Type":"ContainerDied","Data":"fd64c1875ad870b871b16c1ce664e7155bc373a52b92f1240292e23b4f1e7077"} Sep 11 18:50:44.917026 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:44.917006 2556 scope.go:117] "RemoveContainer" containerID="fd64c1875ad870b871b16c1ce664e7155bc373a52b92f1240292e23b4f1e7077" Sep 11 18:50:45.923096 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:45.923058 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-kxkhf" event={"ID":"9462a6cd-3ef7-4962-aaea-01017510998d","Type":"ContainerStarted","Data":"634e4ca6c3bcd0f8d57df137ccbdc6ca3550fc0660d632c5f746273a1529d227"} Sep 11 18:50:46.089052 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:46.089027 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g7ldv_ad0303b1-5952-4ffc-b1d1-d256b97ba1e6/serve-healthcheck-canary/0.log" Sep 11 18:50:46.301928 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:46.301896 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:46.402764 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:46.402730 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:55.742285 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:55.742235 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:55.913060 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:55.913024 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:50:55.997356 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:50:55.997265 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:01.275316 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:51:01.275283 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:01.344189 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:51:01.344151 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:02.098442 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:51:02.098412 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:51:02.102069 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:51:02.102043 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-57dc7f4fdd-72fr6" Sep 11 18:51:23.690858 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:51:23.690823 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:51:23.708793 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:51:23.708764 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:51:24.092873 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:51:24.092845 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 18:51:33.148977 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:51:33.148933 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:33.249234 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:51:33.249199 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:45.012793 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:51:45.012753 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:45.073767 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:51:45.073735 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:46.494492 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:51:46.494450 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:47.859811 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:51:47.859775 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:57.161930 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:51:57.161893 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:57.262242 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:51:57.262213 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:51:57.325211 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:51:57.325183 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:01.289835 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:01.289802 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:01.390597 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:01.390536 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:16.087425 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:16.087395 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:16.155957 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:16.155924 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:19.988900 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:19.988827 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:20.097727 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:20.097665 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:42.648879 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:42.648836 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:43.054921 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:43.054870 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:47.210755 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:47.210714 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:47.307664 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:47.307626 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:48.582406 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:48.582366 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:48.711629 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:48.711594 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:49.520339 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:49.520309 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:49.628866 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:49.628829 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:51.886840 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:51.886804 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:52.004984 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:52.004948 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:52.088624 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:52.088589 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:52.424530 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:52.424470 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:52.502717 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:52.502679 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:59.433964 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:59.433930 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:59.565144 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:59.565112 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:52:59.660340 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:52:59.660303 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:01.035226 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.035195 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-v4224"] Sep 11 18:53:01.035589 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.035536 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c52ab3a-81f5-406d-b050-d9d35ea5ba38" containerName="registry" Sep 11 18:53:01.035589 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.035559 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c52ab3a-81f5-406d-b050-d9d35ea5ba38" containerName="registry" Sep 11 18:53:01.035662 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.035641 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c52ab3a-81f5-406d-b050-d9d35ea5ba38" containerName="registry" Sep 11 18:53:01.038686 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.038669 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-v4224" Sep 11 18:53:01.040814 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.040788 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 18:53:01.041386 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.041356 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6mczg\"" Sep 11 18:53:01.041489 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.041362 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 18:53:01.047704 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.047675 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-v4224"] Sep 11 18:53:01.154419 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.154395 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/954a8b10-b66c-499a-8813-36ea8f515316-bound-sa-token\") pod \"cert-manager-55b4d954d5-v4224\" (UID: \"954a8b10-b66c-499a-8813-36ea8f515316\") " pod="cert-manager/cert-manager-55b4d954d5-v4224" Sep 11 18:53:01.154518 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.154428 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwqp9\" (UniqueName: \"kubernetes.io/projected/954a8b10-b66c-499a-8813-36ea8f515316-kube-api-access-vwqp9\") pod \"cert-manager-55b4d954d5-v4224\" (UID: \"954a8b10-b66c-499a-8813-36ea8f515316\") " pod="cert-manager/cert-manager-55b4d954d5-v4224" Sep 11 18:53:01.254893 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.254859 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/954a8b10-b66c-499a-8813-36ea8f515316-bound-sa-token\") pod \"cert-manager-55b4d954d5-v4224\" (UID: \"954a8b10-b66c-499a-8813-36ea8f515316\") " pod="cert-manager/cert-manager-55b4d954d5-v4224" Sep 11 18:53:01.255055 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.254897 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwqp9\" (UniqueName: \"kubernetes.io/projected/954a8b10-b66c-499a-8813-36ea8f515316-kube-api-access-vwqp9\") pod \"cert-manager-55b4d954d5-v4224\" (UID: \"954a8b10-b66c-499a-8813-36ea8f515316\") " pod="cert-manager/cert-manager-55b4d954d5-v4224" Sep 11 18:53:01.265590 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.265558 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/954a8b10-b66c-499a-8813-36ea8f515316-bound-sa-token\") pod \"cert-manager-55b4d954d5-v4224\" (UID: \"954a8b10-b66c-499a-8813-36ea8f515316\") " pod="cert-manager/cert-manager-55b4d954d5-v4224" Sep 11 18:53:01.268388 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.268363 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwqp9\" (UniqueName: \"kubernetes.io/projected/954a8b10-b66c-499a-8813-36ea8f515316-kube-api-access-vwqp9\") pod \"cert-manager-55b4d954d5-v4224\" (UID: \"954a8b10-b66c-499a-8813-36ea8f515316\") " pod="cert-manager/cert-manager-55b4d954d5-v4224" Sep 11 18:53:01.361631 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.361522 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-v4224" Sep 11 18:53:01.634070 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:01.633997 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-v4224"] Sep 11 18:53:01.636987 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:53:01.636964 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod954a8b10_b66c_499a_8813_36ea8f515316.slice/crio-3ee58e69e9683a5e9f6ca20f5dab7c8a7114cbcc2bbcb62a51e8f7cbff94eaef WatchSource:0}: Error finding container 3ee58e69e9683a5e9f6ca20f5dab7c8a7114cbcc2bbcb62a51e8f7cbff94eaef: Status 404 returned error can't find the container with id 3ee58e69e9683a5e9f6ca20f5dab7c8a7114cbcc2bbcb62a51e8f7cbff94eaef Sep 11 18:53:02.457898 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:02.457826 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-v4224" event={"ID":"954a8b10-b66c-499a-8813-36ea8f515316","Type":"ContainerStarted","Data":"3ee58e69e9683a5e9f6ca20f5dab7c8a7114cbcc2bbcb62a51e8f7cbff94eaef"} Sep 11 18:53:05.083707 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:05.083654 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:05.185149 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:05.185118 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:05.473200 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:05.473121 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-v4224" event={"ID":"954a8b10-b66c-499a-8813-36ea8f515316","Type":"ContainerStarted","Data":"b585af19e6326364950a77677d4cf2fac664cc8d8a48f6b1fcf63d44609ab0ce"} Sep 11 18:53:05.492276 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:05.492232 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-v4224" podStartSLOduration=1.5778594 podStartE2EDuration="4.492217358s" totalImagesPullingTime="2.914357958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:53:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:53:05.491192898 +0000 UTC m=+290.944998853" watchObservedRunningTime="2026-09-11 18:53:05.492217358 +0000 UTC m=+290.946023313" Sep 11 18:53:07.446359 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:07.446322 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:14.972073 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:14.972044 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 18:53:14.983680 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:14.983653 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 18:53:14.986086 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:14.986056 2556 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 18:53:17.511361 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:17.511322 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:17.622045 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:17.622001 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:22.351065 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:22.351023 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:22.470107 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:22.470068 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:22.643191 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:22.643121 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:22.999556 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:22.999457 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:23.265088 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.265021 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:23.391493 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.391454 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:23.495559 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.495513 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5"] Sep 11 18:53:23.499003 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.498984 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.502662 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.502637 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 18:53:23.502770 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.502680 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 18:53:23.502770 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.502716 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-snlds\"" Sep 11 18:53:23.502770 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.502732 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 18:53:23.502770 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.502739 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 18:53:23.528050 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.527996 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/94668d09-5d8d-4a84-a81e-6d6571619c58-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c9988978f-8s8t5\" (UID: \"94668d09-5d8d-4a84-a81e-6d6571619c58\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.528050 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.528026 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94668d09-5d8d-4a84-a81e-6d6571619c58-tmp\") pod \"opendatahub-operator-controller-manager-c9988978f-8s8t5\" (UID: \"94668d09-5d8d-4a84-a81e-6d6571619c58\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.528174 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.528057 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94668d09-5d8d-4a84-a81e-6d6571619c58-webhook-cert\") pod \"opendatahub-operator-controller-manager-c9988978f-8s8t5\" (UID: \"94668d09-5d8d-4a84-a81e-6d6571619c58\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.528174 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.528127 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkp4k\" (UniqueName: \"kubernetes.io/projected/94668d09-5d8d-4a84-a81e-6d6571619c58-kube-api-access-rkp4k\") pod \"opendatahub-operator-controller-manager-c9988978f-8s8t5\" (UID: \"94668d09-5d8d-4a84-a81e-6d6571619c58\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.528174 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.528160 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/94668d09-5d8d-4a84-a81e-6d6571619c58-manifests\") pod \"opendatahub-operator-controller-manager-c9988978f-8s8t5\" (UID: \"94668d09-5d8d-4a84-a81e-6d6571619c58\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.585845 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.585808 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5"] Sep 11 18:53:23.629572 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.629528 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/94668d09-5d8d-4a84-a81e-6d6571619c58-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c9988978f-8s8t5\" (UID: \"94668d09-5d8d-4a84-a81e-6d6571619c58\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.629696 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.629579 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94668d09-5d8d-4a84-a81e-6d6571619c58-tmp\") pod \"opendatahub-operator-controller-manager-c9988978f-8s8t5\" (UID: \"94668d09-5d8d-4a84-a81e-6d6571619c58\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.629696 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.629637 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94668d09-5d8d-4a84-a81e-6d6571619c58-webhook-cert\") pod \"opendatahub-operator-controller-manager-c9988978f-8s8t5\" (UID: \"94668d09-5d8d-4a84-a81e-6d6571619c58\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.629696 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.629690 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkp4k\" (UniqueName: \"kubernetes.io/projected/94668d09-5d8d-4a84-a81e-6d6571619c58-kube-api-access-rkp4k\") pod \"opendatahub-operator-controller-manager-c9988978f-8s8t5\" (UID: \"94668d09-5d8d-4a84-a81e-6d6571619c58\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.629858 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.629732 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/94668d09-5d8d-4a84-a81e-6d6571619c58-manifests\") pod \"opendatahub-operator-controller-manager-c9988978f-8s8t5\" (UID: \"94668d09-5d8d-4a84-a81e-6d6571619c58\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.630082 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.630061 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/94668d09-5d8d-4a84-a81e-6d6571619c58-manifests\") pod \"opendatahub-operator-controller-manager-c9988978f-8s8t5\" (UID: \"94668d09-5d8d-4a84-a81e-6d6571619c58\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.631823 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.631803 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94668d09-5d8d-4a84-a81e-6d6571619c58-tmp\") pod \"opendatahub-operator-controller-manager-c9988978f-8s8t5\" (UID: \"94668d09-5d8d-4a84-a81e-6d6571619c58\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.632033 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.632017 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94668d09-5d8d-4a84-a81e-6d6571619c58-webhook-cert\") pod \"opendatahub-operator-controller-manager-c9988978f-8s8t5\" (UID: \"94668d09-5d8d-4a84-a81e-6d6571619c58\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.632188 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.632168 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/94668d09-5d8d-4a84-a81e-6d6571619c58-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c9988978f-8s8t5\" (UID: \"94668d09-5d8d-4a84-a81e-6d6571619c58\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.639793 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.639765 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkp4k\" (UniqueName: \"kubernetes.io/projected/94668d09-5d8d-4a84-a81e-6d6571619c58-kube-api-access-rkp4k\") pod \"opendatahub-operator-controller-manager-c9988978f-8s8t5\" (UID: \"94668d09-5d8d-4a84-a81e-6d6571619c58\") " pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.661642 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.661620 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:23.760861 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.760832 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:23.808770 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.808738 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:23.861701 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.861671 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:23.964933 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:23.964892 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:24.070287 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:24.070199 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:24.090145 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:24.090126 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5"] Sep 11 18:53:24.092977 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:53:24.092947 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94668d09_5d8d_4a84_a81e_6d6571619c58.slice/crio-1d19aa98c2dd7a27f514fb0b9033bcfad2b3398ad6ee911efd9520d939ed7435 WatchSource:0}: Error finding container 1d19aa98c2dd7a27f514fb0b9033bcfad2b3398ad6ee911efd9520d939ed7435: Status 404 returned error can't find the container with id 1d19aa98c2dd7a27f514fb0b9033bcfad2b3398ad6ee911efd9520d939ed7435 Sep 11 18:53:24.095236 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:24.095215 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:53:24.159552 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:24.159511 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:24.547608 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:24.547571 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" event={"ID":"94668d09-5d8d-4a84-a81e-6d6571619c58","Type":"ContainerStarted","Data":"1d19aa98c2dd7a27f514fb0b9033bcfad2b3398ad6ee911efd9520d939ed7435"} Sep 11 18:53:27.566624 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:27.566593 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" event={"ID":"94668d09-5d8d-4a84-a81e-6d6571619c58","Type":"ContainerStarted","Data":"315bacd4c32e03e19367ccb71ac2bcc666fdfbb28a5f9c9a284f12f113661d6d"} Sep 11 18:53:28.572972 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:28.572599 2556 generic.go:356] "Generic (PLEG): container finished" podID="94668d09-5d8d-4a84-a81e-6d6571619c58" containerID="315bacd4c32e03e19367ccb71ac2bcc666fdfbb28a5f9c9a284f12f113661d6d" exitCode=0 Sep 11 18:53:28.572972 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:28.572708 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" event={"ID":"94668d09-5d8d-4a84-a81e-6d6571619c58","Type":"ContainerDied","Data":"315bacd4c32e03e19367ccb71ac2bcc666fdfbb28a5f9c9a284f12f113661d6d"} Sep 11 18:53:29.579223 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:29.579187 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" event={"ID":"94668d09-5d8d-4a84-a81e-6d6571619c58","Type":"ContainerStarted","Data":"179d6287fb295e9291c461abd7aa45b8f332ce09d352a351aba74186791f9349"} Sep 11 18:53:29.579643 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:29.579300 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:29.600031 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:29.599984 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" podStartSLOduration=2.096909045 podStartE2EDuration="6.599967735s" totalImagesPullingTime="3.417956769s" totalInitContainerRuntime="1.085101921s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:53:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:53:29.598832356 +0000 UTC m=+315.052638311" watchObservedRunningTime="2026-09-11 18:53:29.599967735 +0000 UTC m=+315.053773700" Sep 11 18:53:31.636434 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:31.636393 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:31.738573 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:31.738513 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:40.586612 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:40.586585 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c9988978f-8s8t5" Sep 11 18:53:41.211320 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:41.211283 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:41.309636 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:41.309602 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:42.883625 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:42.883591 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:43.015757 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:43.015717 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:43.428479 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:43.428439 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:43.714802 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:43.714711 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:45.855749 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:45.855717 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:45.948683 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:45.948650 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:47.300217 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:47.300175 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:50.665064 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:50.665033 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:50.765915 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:50.765886 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:51.453442 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:51.453410 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:51.651706 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:51.651670 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:51.764917 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:51.764843 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:54.177522 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:54.177484 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:54.312699 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:54.312667 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:54.408907 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:54.408876 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:54.782703 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:54.782637 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:53:54.879574 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:53:54.879526 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:00.766404 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.766364 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t"] Sep 11 18:54:00.835057 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.835026 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t"] Sep 11 18:54:00.835202 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.835122 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:00.837288 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.837265 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 18:54:00.837446 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.837431 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 11 18:54:00.837516 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.837461 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 11 18:54:00.837600 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.837461 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 11 18:54:00.838745 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.838716 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fr95c\"" Sep 11 18:54:00.838745 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.838740 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 18:54:00.838902 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.838716 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 18:54:00.950629 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.950598 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53506b1c-b55c-4c79-9cd7-f672455e3c08-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:00.950815 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.950645 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53506b1c-b55c-4c79-9cd7-f672455e3c08-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:00.950815 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.950751 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53506b1c-b55c-4c79-9cd7-f672455e3c08-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:00.950815 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.950786 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53506b1c-b55c-4c79-9cd7-f672455e3c08-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:00.950998 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.950860 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53506b1c-b55c-4c79-9cd7-f672455e3c08-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:00.950998 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.950926 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53506b1c-b55c-4c79-9cd7-f672455e3c08-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:00.950998 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:00.950963 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6f5p\" (UniqueName: \"kubernetes.io/projected/53506b1c-b55c-4c79-9cd7-f672455e3c08-kube-api-access-b6f5p\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:01.051941 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.051911 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53506b1c-b55c-4c79-9cd7-f672455e3c08-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:01.052101 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.051955 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53506b1c-b55c-4c79-9cd7-f672455e3c08-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:01.052101 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.051975 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53506b1c-b55c-4c79-9cd7-f672455e3c08-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:01.052101 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.051999 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53506b1c-b55c-4c79-9cd7-f672455e3c08-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:01.052101 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.052031 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53506b1c-b55c-4c79-9cd7-f672455e3c08-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:01.052101 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.052052 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6f5p\" (UniqueName: \"kubernetes.io/projected/53506b1c-b55c-4c79-9cd7-f672455e3c08-kube-api-access-b6f5p\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:01.052101 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.052076 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53506b1c-b55c-4c79-9cd7-f672455e3c08-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:01.052941 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.052911 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53506b1c-b55c-4c79-9cd7-f672455e3c08-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:01.054307 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.054281 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53506b1c-b55c-4c79-9cd7-f672455e3c08-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:01.054470 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.054451 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53506b1c-b55c-4c79-9cd7-f672455e3c08-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:01.054672 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.054656 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53506b1c-b55c-4c79-9cd7-f672455e3c08-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:01.054747 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.054731 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53506b1c-b55c-4c79-9cd7-f672455e3c08-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:01.063410 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.063388 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53506b1c-b55c-4c79-9cd7-f672455e3c08-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:01.064602 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.064579 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6f5p\" (UniqueName: \"kubernetes.io/projected/53506b1c-b55c-4c79-9cd7-f672455e3c08-kube-api-access-b6f5p\") pod \"istiod-openshift-gateway-5677ccb49b-6m92t\" (UID: \"53506b1c-b55c-4c79-9cd7-f672455e3c08\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:01.145029 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.145004 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:01.186660 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.186095 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf"] Sep 11 18:54:01.190189 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.190080 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" Sep 11 18:54:01.194558 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.194519 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 11 18:54:01.194906 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.194886 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mmf2p\"" Sep 11 18:54:01.195221 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.195204 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 11 18:54:01.195603 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.195585 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 11 18:54:01.195911 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.195895 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 11 18:54:01.196284 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.196266 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:54:01.214674 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.214646 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf"] Sep 11 18:54:01.355454 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.355368 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ba3b8052-b4d5-42e7-a609-dd8ed95b8d28-manager-config\") pod \"lws-controller-manager-744bc9d7dd-wbcxf\" (UID: \"ba3b8052-b4d5-42e7-a609-dd8ed95b8d28\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" Sep 11 18:54:01.355454 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.355411 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsk59\" (UniqueName: \"kubernetes.io/projected/ba3b8052-b4d5-42e7-a609-dd8ed95b8d28-kube-api-access-tsk59\") pod \"lws-controller-manager-744bc9d7dd-wbcxf\" (UID: \"ba3b8052-b4d5-42e7-a609-dd8ed95b8d28\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" Sep 11 18:54:01.355454 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.355451 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba3b8052-b4d5-42e7-a609-dd8ed95b8d28-metrics-cert\") pod \"lws-controller-manager-744bc9d7dd-wbcxf\" (UID: \"ba3b8052-b4d5-42e7-a609-dd8ed95b8d28\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" Sep 11 18:54:01.355672 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.355474 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba3b8052-b4d5-42e7-a609-dd8ed95b8d28-cert\") pod \"lws-controller-manager-744bc9d7dd-wbcxf\" (UID: \"ba3b8052-b4d5-42e7-a609-dd8ed95b8d28\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" Sep 11 18:54:01.456501 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.456470 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ba3b8052-b4d5-42e7-a609-dd8ed95b8d28-manager-config\") pod \"lws-controller-manager-744bc9d7dd-wbcxf\" (UID: \"ba3b8052-b4d5-42e7-a609-dd8ed95b8d28\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" Sep 11 18:54:01.456669 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.456514 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsk59\" (UniqueName: \"kubernetes.io/projected/ba3b8052-b4d5-42e7-a609-dd8ed95b8d28-kube-api-access-tsk59\") pod \"lws-controller-manager-744bc9d7dd-wbcxf\" (UID: \"ba3b8052-b4d5-42e7-a609-dd8ed95b8d28\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" Sep 11 18:54:01.456669 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.456569 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba3b8052-b4d5-42e7-a609-dd8ed95b8d28-metrics-cert\") pod \"lws-controller-manager-744bc9d7dd-wbcxf\" (UID: \"ba3b8052-b4d5-42e7-a609-dd8ed95b8d28\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" Sep 11 18:54:01.456669 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.456595 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba3b8052-b4d5-42e7-a609-dd8ed95b8d28-cert\") pod \"lws-controller-manager-744bc9d7dd-wbcxf\" (UID: \"ba3b8052-b4d5-42e7-a609-dd8ed95b8d28\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" Sep 11 18:54:01.457073 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.457052 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ba3b8052-b4d5-42e7-a609-dd8ed95b8d28-manager-config\") pod \"lws-controller-manager-744bc9d7dd-wbcxf\" (UID: \"ba3b8052-b4d5-42e7-a609-dd8ed95b8d28\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" Sep 11 18:54:01.458924 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.458898 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba3b8052-b4d5-42e7-a609-dd8ed95b8d28-cert\") pod \"lws-controller-manager-744bc9d7dd-wbcxf\" (UID: \"ba3b8052-b4d5-42e7-a609-dd8ed95b8d28\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" Sep 11 18:54:01.459045 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.459021 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba3b8052-b4d5-42e7-a609-dd8ed95b8d28-metrics-cert\") pod \"lws-controller-manager-744bc9d7dd-wbcxf\" (UID: \"ba3b8052-b4d5-42e7-a609-dd8ed95b8d28\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" Sep 11 18:54:01.467725 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.467704 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsk59\" (UniqueName: \"kubernetes.io/projected/ba3b8052-b4d5-42e7-a609-dd8ed95b8d28-kube-api-access-tsk59\") pod \"lws-controller-manager-744bc9d7dd-wbcxf\" (UID: \"ba3b8052-b4d5-42e7-a609-dd8ed95b8d28\") " pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" Sep 11 18:54:01.473650 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.473628 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t"] Sep 11 18:54:01.479583 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:54:01.479560 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53506b1c_b55c_4c79_9cd7_f672455e3c08.slice/crio-39fe6745ac376a9de644aa8dc03fb40308a40b2c132a3a6abfe63d7a3e601b3a WatchSource:0}: Error finding container 39fe6745ac376a9de644aa8dc03fb40308a40b2c132a3a6abfe63d7a3e601b3a: Status 404 returned error can't find the container with id 39fe6745ac376a9de644aa8dc03fb40308a40b2c132a3a6abfe63d7a3e601b3a Sep 11 18:54:01.518460 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.517599 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" Sep 11 18:54:01.722354 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.722276 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" event={"ID":"53506b1c-b55c-4c79-9cd7-f672455e3c08","Type":"ContainerStarted","Data":"39fe6745ac376a9de644aa8dc03fb40308a40b2c132a3a6abfe63d7a3e601b3a"} Sep 11 18:54:01.810014 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:01.809980 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf"] Sep 11 18:54:01.812910 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:54:01.812883 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3b8052_b4d5_42e7_a609_dd8ed95b8d28.slice/crio-362113fa75831e725a7e1ddcbc50d03e5ce4ddb3fbb8653c646da9d0e2203b38 WatchSource:0}: Error finding container 362113fa75831e725a7e1ddcbc50d03e5ce4ddb3fbb8653c646da9d0e2203b38: Status 404 returned error can't find the container with id 362113fa75831e725a7e1ddcbc50d03e5ce4ddb3fbb8653c646da9d0e2203b38 Sep 11 18:54:02.731121 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:02.731043 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" event={"ID":"ba3b8052-b4d5-42e7-a609-dd8ed95b8d28","Type":"ContainerStarted","Data":"362113fa75831e725a7e1ddcbc50d03e5ce4ddb3fbb8653c646da9d0e2203b38"} Sep 11 18:54:04.996796 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:04.996760 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 18:54:04.997046 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:04.996853 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 18:54:05.269230 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:05.269140 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:05.466381 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:05.466345 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:05.667298 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:05.667264 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:05.748161 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:05.748127 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" event={"ID":"53506b1c-b55c-4c79-9cd7-f672455e3c08","Type":"ContainerStarted","Data":"ee65c29507a9993c9064b6e21f133371c8539cfb6e4fefb0d1a2a23dd9f8ce06"} Sep 11 18:54:05.748368 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:05.748304 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:05.750042 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:05.750018 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" event={"ID":"ba3b8052-b4d5-42e7-a609-dd8ed95b8d28","Type":"ContainerStarted","Data":"722620f09e98007e0afe334f0d1ee2e235d603bebd3d154043185730f27ab6a7"} Sep 11 18:54:05.750166 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:05.750158 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" Sep 11 18:54:05.804683 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:05.804623 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" podStartSLOduration=2.28966888 podStartE2EDuration="5.804607362s" totalImagesPullingTime="3.514938482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:54:05.803904347 +0000 UTC m=+351.257710324" watchObservedRunningTime="2026-09-11 18:54:05.804607362 +0000 UTC m=+351.258413317" Sep 11 18:54:05.824353 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:05.824322 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:05.860304 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:05.860258 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" podStartSLOduration=1.624314417 podStartE2EDuration="4.860241906s" totalImagesPullingTime="3.235927489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:54:05.85724462 +0000 UTC m=+351.311050594" watchObservedRunningTime="2026-09-11 18:54:05.860241906 +0000 UTC m=+351.314047861" Sep 11 18:54:06.060197 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:06.060159 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:06.756166 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:06.756127 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6m92t" Sep 11 18:54:06.886769 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:06.886737 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:07.112748 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:07.112716 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:07.214663 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:07.214567 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:16.756881 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:16.756850 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-744bc9d7dd-wbcxf" Sep 11 18:54:54.957978 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:54.957945 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:54:56.075654 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:54:56.075590 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:19.717176 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:19.717145 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:21.657276 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:21.657237 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:21.740501 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:21.740473 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:22.312370 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:22.312325 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:22.716741 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:22.716664 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:24.969403 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:24.969367 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:25.130174 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:25.130143 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:26.358324 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:26.358290 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:27.374650 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:27.374618 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:27.591875 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:27.591842 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:27.711618 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:27.711531 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:28.408703 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:28.408667 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:28.622766 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:28.622734 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:28.777884 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:28.777802 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:32.328028 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:32.327994 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:32.444894 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:32.444860 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:32.542788 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:32.542761 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:33.540811 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:33.540772 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:33.639945 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:33.639910 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:37.146219 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:37.146153 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:37.250355 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:37.250285 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:39.184846 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:39.184806 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:39.671883 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:39.671853 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:39.767686 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:39.767654 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:40.008230 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:40.008160 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:46.154978 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:46.154938 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:46.249865 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:46.249822 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:46.353593 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:46.353560 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:46.452519 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:46.452443 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:51.087933 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:51.087901 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh"] Sep 11 18:55:51.090362 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:51.090340 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" Sep 11 18:55:51.092774 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:51.092752 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 18:55:51.092872 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:51.092795 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 18:55:51.093138 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:51.093121 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r9t7x\"" Sep 11 18:55:51.111682 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:51.111663 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh"] Sep 11 18:55:51.171405 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:51.171382 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a09e5f0f-c582-4d68-a7b5-b8599e41b283-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frrkh\" (UID: \"a09e5f0f-c582-4d68-a7b5-b8599e41b283\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" Sep 11 18:55:51.171520 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:51.171428 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmxx2\" (UniqueName: \"kubernetes.io/projected/a09e5f0f-c582-4d68-a7b5-b8599e41b283-kube-api-access-pmxx2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frrkh\" (UID: \"a09e5f0f-c582-4d68-a7b5-b8599e41b283\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" Sep 11 18:55:51.272245 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:51.272211 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmxx2\" (UniqueName: \"kubernetes.io/projected/a09e5f0f-c582-4d68-a7b5-b8599e41b283-kube-api-access-pmxx2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frrkh\" (UID: \"a09e5f0f-c582-4d68-a7b5-b8599e41b283\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" Sep 11 18:55:51.272405 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:51.272356 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a09e5f0f-c582-4d68-a7b5-b8599e41b283-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frrkh\" (UID: \"a09e5f0f-c582-4d68-a7b5-b8599e41b283\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" Sep 11 18:55:51.272711 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:51.272694 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a09e5f0f-c582-4d68-a7b5-b8599e41b283-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frrkh\" (UID: \"a09e5f0f-c582-4d68-a7b5-b8599e41b283\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" Sep 11 18:55:51.292995 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:51.292955 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmxx2\" (UniqueName: \"kubernetes.io/projected/a09e5f0f-c582-4d68-a7b5-b8599e41b283-kube-api-access-pmxx2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-frrkh\" (UID: \"a09e5f0f-c582-4d68-a7b5-b8599e41b283\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" Sep 11 18:55:51.400677 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:51.400601 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" Sep 11 18:55:51.680486 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:51.680404 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh"] Sep 11 18:55:51.684636 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:55:51.684614 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09e5f0f_c582_4d68_a7b5_b8599e41b283.slice/crio-08d751ec6b29f2388bde37b146a4909a50d36b2cd404b72f0e49dfa6943d8c33 WatchSource:0}: Error finding container 08d751ec6b29f2388bde37b146a4909a50d36b2cd404b72f0e49dfa6943d8c33: Status 404 returned error can't find the container with id 08d751ec6b29f2388bde37b146a4909a50d36b2cd404b72f0e49dfa6943d8c33 Sep 11 18:55:52.207505 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:52.207466 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" event={"ID":"a09e5f0f-c582-4d68-a7b5-b8599e41b283","Type":"ContainerStarted","Data":"08d751ec6b29f2388bde37b146a4909a50d36b2cd404b72f0e49dfa6943d8c33"} Sep 11 18:55:52.894860 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:52.894811 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:53.000716 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:53.000664 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:56.235372 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:56.235338 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" event={"ID":"a09e5f0f-c582-4d68-a7b5-b8599e41b283","Type":"ContainerStarted","Data":"7ab679f88706cca4e170178157bf6d818904f444e290652f3dd6c7d13ead7464"} Sep 11 18:55:56.235881 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:56.235435 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" Sep 11 18:55:56.707856 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:56.707817 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:56.931595 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:56.931558 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:55:57.129705 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:55:57.129673 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:00.582575 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:00.582524 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:00.654949 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:00.654920 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:05.915181 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:05.915145 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:06.016945 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:06.016916 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:06.330335 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:06.330296 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:06.430283 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:06.430244 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:06.497737 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:06.497698 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:07.242209 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:07.242182 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" Sep 11 18:56:07.270996 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:07.270944 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" podStartSLOduration=12.753846498 podStartE2EDuration="16.270922841s" totalImagesPullingTime="3.517076343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:55:56.268559254 +0000 UTC m=+461.722365221" watchObservedRunningTime="2026-09-11 18:56:07.270922841 +0000 UTC m=+472.724728798" Sep 11 18:56:23.628958 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:23.628923 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:23.834311 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:23.834280 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:26.221605 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:26.221566 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:28.421087 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:28.421045 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:28.535883 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:28.535837 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:28.633431 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:28.633397 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:29.269213 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:29.269163 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:29.356914 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:29.356884 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:44.483466 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:44.483430 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:44.583253 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:44.583211 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:44.920349 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:44.920317 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qckvx"] Sep 11 18:56:44.922591 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:44.922569 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qckvx" Sep 11 18:56:44.924785 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:44.924760 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 18:56:44.924898 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:44.924760 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 18:56:44.924898 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:44.924767 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 11 18:56:44.925383 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:44.925366 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-sch9r\"" Sep 11 18:56:44.928430 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:44.928405 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsmxc\" (UniqueName: \"kubernetes.io/projected/1f4f6a7e-46be-42f3-bd1c-adc4e72b3060-kube-api-access-wsmxc\") pod \"postgres-69fb4594cf-qckvx\" (UID: \"1f4f6a7e-46be-42f3-bd1c-adc4e72b3060\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qckvx" Sep 11 18:56:44.928608 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:44.928586 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1f4f6a7e-46be-42f3-bd1c-adc4e72b3060-data\") pod \"postgres-69fb4594cf-qckvx\" (UID: \"1f4f6a7e-46be-42f3-bd1c-adc4e72b3060\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qckvx" Sep 11 18:56:44.932259 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:44.932239 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qckvx"] Sep 11 18:56:44.958263 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:44.958234 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:45.030160 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:45.030127 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1f4f6a7e-46be-42f3-bd1c-adc4e72b3060-data\") pod \"postgres-69fb4594cf-qckvx\" (UID: \"1f4f6a7e-46be-42f3-bd1c-adc4e72b3060\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qckvx" Sep 11 18:56:45.030332 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:45.030194 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsmxc\" (UniqueName: \"kubernetes.io/projected/1f4f6a7e-46be-42f3-bd1c-adc4e72b3060-kube-api-access-wsmxc\") pod \"postgres-69fb4594cf-qckvx\" (UID: \"1f4f6a7e-46be-42f3-bd1c-adc4e72b3060\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qckvx" Sep 11 18:56:45.030620 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:45.030595 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1f4f6a7e-46be-42f3-bd1c-adc4e72b3060-data\") pod \"postgres-69fb4594cf-qckvx\" (UID: \"1f4f6a7e-46be-42f3-bd1c-adc4e72b3060\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qckvx" Sep 11 18:56:45.038251 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:45.038225 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsmxc\" (UniqueName: \"kubernetes.io/projected/1f4f6a7e-46be-42f3-bd1c-adc4e72b3060-kube-api-access-wsmxc\") pod \"postgres-69fb4594cf-qckvx\" (UID: \"1f4f6a7e-46be-42f3-bd1c-adc4e72b3060\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qckvx" Sep 11 18:56:45.234810 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:45.234726 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qckvx" Sep 11 18:56:45.502240 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:45.502170 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qckvx"] Sep 11 18:56:45.505893 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:56:45.505863 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4f6a7e_46be_42f3_bd1c_adc4e72b3060.slice/crio-33be514a7fcf1d27a020e21bb96d97eceaca86d0b08eacb4cee3a8a2eb05f8cf WatchSource:0}: Error finding container 33be514a7fcf1d27a020e21bb96d97eceaca86d0b08eacb4cee3a8a2eb05f8cf: Status 404 returned error can't find the container with id 33be514a7fcf1d27a020e21bb96d97eceaca86d0b08eacb4cee3a8a2eb05f8cf Sep 11 18:56:46.458219 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:46.458186 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qckvx" event={"ID":"1f4f6a7e-46be-42f3-bd1c-adc4e72b3060","Type":"ContainerStarted","Data":"33be514a7fcf1d27a020e21bb96d97eceaca86d0b08eacb4cee3a8a2eb05f8cf"} Sep 11 18:56:51.513595 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:51.513557 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qckvx" event={"ID":"1f4f6a7e-46be-42f3-bd1c-adc4e72b3060","Type":"ContainerStarted","Data":"a5f0e01b4fa30454a6c66069de1b2840b655392e977f15f34413be33a1c3211e"} Sep 11 18:56:51.514075 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:51.513622 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qckvx" Sep 11 18:56:51.530820 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:51.530761 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qckvx" podStartSLOduration=2.117062171 podStartE2EDuration="7.530744556s" totalImagesPullingTime="5.413682385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:56:51.528839663 +0000 UTC m=+516.982645642" watchObservedRunningTime="2026-09-11 18:56:51.530744556 +0000 UTC m=+516.984550511" Sep 11 18:56:57.548258 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:57.548223 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qckvx" Sep 11 18:56:57.846470 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:57.846381 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:56:57.968883 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:56:57.968848 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:05.426755 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:05.426722 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:05.525225 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:05.525195 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:05.627093 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:05.627045 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:06.211273 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:06.211237 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:09.373154 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:09.373069 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:09.461227 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:09.461195 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:40.895044 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:40.895011 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:40.994996 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:40.994958 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:42.345718 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:42.345682 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:42.436146 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:42.436100 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:42.530014 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:42.529986 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:42.638686 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:42.638616 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:42.875908 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:42.875872 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:42.977606 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:42.977510 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:42.984607 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:42.984579 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-k9fdk"] Sep 11 18:57:42.987944 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:42.987921 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k9fdk" Sep 11 18:57:42.990383 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:42.990364 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 11 18:57:43.000245 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:43.000224 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-k9fdk"] Sep 11 18:57:43.013570 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:43.013528 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx76m\" (UniqueName: \"kubernetes.io/projected/929571ec-dc93-4566-b194-c2b97d95a535-kube-api-access-nx76m\") pod \"payload-pre-processing-59fbc47bf8-k9fdk\" (UID: \"929571ec-dc93-4566-b194-c2b97d95a535\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k9fdk" Sep 11 18:57:43.013682 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:43.013665 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929571ec-dc93-4566-b194-c2b97d95a535-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-k9fdk\" (UID: \"929571ec-dc93-4566-b194-c2b97d95a535\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k9fdk" Sep 11 18:57:43.114159 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:43.114124 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx76m\" (UniqueName: \"kubernetes.io/projected/929571ec-dc93-4566-b194-c2b97d95a535-kube-api-access-nx76m\") pod \"payload-pre-processing-59fbc47bf8-k9fdk\" (UID: \"929571ec-dc93-4566-b194-c2b97d95a535\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k9fdk" Sep 11 18:57:43.114339 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:43.114227 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929571ec-dc93-4566-b194-c2b97d95a535-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-k9fdk\" (UID: \"929571ec-dc93-4566-b194-c2b97d95a535\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k9fdk" Sep 11 18:57:43.115094 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:43.115070 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929571ec-dc93-4566-b194-c2b97d95a535-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-k9fdk\" (UID: \"929571ec-dc93-4566-b194-c2b97d95a535\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k9fdk" Sep 11 18:57:43.123447 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:43.123418 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx76m\" (UniqueName: \"kubernetes.io/projected/929571ec-dc93-4566-b194-c2b97d95a535-kube-api-access-nx76m\") pod \"payload-pre-processing-59fbc47bf8-k9fdk\" (UID: \"929571ec-dc93-4566-b194-c2b97d95a535\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k9fdk" Sep 11 18:57:43.298597 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:43.298562 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k9fdk" Sep 11 18:57:43.571898 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:43.571871 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-k9fdk"] Sep 11 18:57:43.574036 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:57:43.574009 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929571ec_dc93_4566_b194_c2b97d95a535.slice/crio-d6e3088c65eb85a597cf33b808565a5550edcaf28538fd13c13e73b236952a42 WatchSource:0}: Error finding container d6e3088c65eb85a597cf33b808565a5550edcaf28538fd13c13e73b236952a42: Status 404 returned error can't find the container with id d6e3088c65eb85a597cf33b808565a5550edcaf28538fd13c13e73b236952a42 Sep 11 18:57:43.761435 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:43.761405 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k9fdk" event={"ID":"929571ec-dc93-4566-b194-c2b97d95a535","Type":"ContainerStarted","Data":"d6e3088c65eb85a597cf33b808565a5550edcaf28538fd13c13e73b236952a42"} Sep 11 18:57:45.772681 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:45.772648 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k9fdk" event={"ID":"929571ec-dc93-4566-b194-c2b97d95a535","Type":"ContainerStarted","Data":"92b51c60612c16074f2d0fc6aab15b3835db167629842d98a621362978913638"} Sep 11 18:57:45.773114 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:45.772775 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k9fdk" Sep 11 18:57:45.789403 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:45.789347 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k9fdk" podStartSLOduration=2.469314581 podStartE2EDuration="3.789332254s" totalImagesPullingTime="1.320017673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:57:45.786914104 +0000 UTC m=+571.240720072" watchObservedRunningTime="2026-09-11 18:57:45.789332254 +0000 UTC m=+571.243138210" Sep 11 18:57:52.919255 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:52.919223 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:53.007066 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:53.007034 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:57:56.779128 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:57:56.779099 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k9fdk" Sep 11 18:58:03.961315 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:03.961280 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:58:15.009551 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:15.009509 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 18:58:15.012494 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:15.012470 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 18:58:31.054262 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:31.054230 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:58:41.069698 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:41.069621 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:58:41.170873 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:41.170842 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:58:47.101876 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:47.101847 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:58:47.210884 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:47.210857 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:58:51.238897 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.238862 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs"] Sep 11 18:58:51.242473 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.242452 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.245207 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.245184 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 18:58:51.245317 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.245212 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tgc84\"" Sep 11 18:58:51.245317 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.245234 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 11 18:58:51.245317 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.245257 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 18:58:51.252228 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.252205 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs"] Sep 11 18:58:51.285906 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.285878 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c54d6285-3940-44e1-ac75-876677a401cb-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.286048 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.285919 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c54d6285-3940-44e1-ac75-876677a401cb-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.286048 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.285947 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c54d6285-3940-44e1-ac75-876677a401cb-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.286048 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.285991 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vq86\" (UniqueName: \"kubernetes.io/projected/c54d6285-3940-44e1-ac75-876677a401cb-kube-api-access-8vq86\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.286167 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.286098 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c54d6285-3940-44e1-ac75-876677a401cb-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.286167 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.286127 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c54d6285-3940-44e1-ac75-876677a401cb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.334284 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.334236 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:58:51.387278 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.387245 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c54d6285-3940-44e1-ac75-876677a401cb-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.387278 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.387277 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c54d6285-3940-44e1-ac75-876677a401cb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.387451 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.387353 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c54d6285-3940-44e1-ac75-876677a401cb-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.387451 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.387382 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c54d6285-3940-44e1-ac75-876677a401cb-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.387451 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.387408 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c54d6285-3940-44e1-ac75-876677a401cb-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.387645 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.387452 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vq86\" (UniqueName: \"kubernetes.io/projected/c54d6285-3940-44e1-ac75-876677a401cb-kube-api-access-8vq86\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.387717 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.387701 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c54d6285-3940-44e1-ac75-876677a401cb-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.387770 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.387751 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c54d6285-3940-44e1-ac75-876677a401cb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.387817 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.387803 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c54d6285-3940-44e1-ac75-876677a401cb-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.389646 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.389628 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c54d6285-3940-44e1-ac75-876677a401cb-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.389831 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.389813 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c54d6285-3940-44e1-ac75-876677a401cb-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.394330 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.394309 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vq86\" (UniqueName: \"kubernetes.io/projected/c54d6285-3940-44e1-ac75-876677a401cb-kube-api-access-8vq86\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs\" (UID: \"c54d6285-3940-44e1-ac75-876677a401cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.553261 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.553225 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:58:51.823133 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.823104 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs"] Sep 11 18:58:51.825737 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:58:51.825707 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54d6285_3940_44e1_ac75_876677a401cb.slice/crio-b63aa39bd9c1fb3a88bb370991de9c41feaefab672f0ed83b4b93144494fed9a WatchSource:0}: Error finding container b63aa39bd9c1fb3a88bb370991de9c41feaefab672f0ed83b4b93144494fed9a: Status 404 returned error can't find the container with id b63aa39bd9c1fb3a88bb370991de9c41feaefab672f0ed83b4b93144494fed9a Sep 11 18:58:51.827072 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:51.827051 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:58:52.086359 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:52.086272 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" event={"ID":"c54d6285-3940-44e1-ac75-876677a401cb","Type":"ContainerStarted","Data":"b63aa39bd9c1fb3a88bb370991de9c41feaefab672f0ed83b4b93144494fed9a"} Sep 11 18:58:53.092909 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:53.092820 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" event={"ID":"c54d6285-3940-44e1-ac75-876677a401cb","Type":"ContainerStarted","Data":"baf32c8a7363a678a1156c48d973c85e41e2d0d99a257fa443f53e3396a6ff51"} Sep 11 18:58:53.110081 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:58:53.110030 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" podStartSLOduration=1.130417896 podStartE2EDuration="2.110013388s" totalImagesPullingTime="979.595492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:58:53.108337204 +0000 UTC m=+638.562143158" watchObservedRunningTime="2026-09-11 18:58:53.110013388 +0000 UTC m=+638.563819361" Sep 11 18:59:00.332085 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.332048 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr"] Sep 11 18:59:00.335709 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.335687 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.341614 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.338341 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 11 18:59:00.347587 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.347049 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr"] Sep 11 18:59:00.365126 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.365100 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:59:00.462689 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.462662 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv4z8\" (UniqueName: \"kubernetes.io/projected/9eedf2a6-9683-46f6-afbc-464ac0e312bc-kube-api-access-gv4z8\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.462689 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.462692 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eedf2a6-9683-46f6-afbc-464ac0e312bc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.462843 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.462716 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eedf2a6-9683-46f6-afbc-464ac0e312bc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.462843 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.462739 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eedf2a6-9683-46f6-afbc-464ac0e312bc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.462843 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.462809 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eedf2a6-9683-46f6-afbc-464ac0e312bc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.462939 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.462844 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eedf2a6-9683-46f6-afbc-464ac0e312bc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.563815 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.563783 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv4z8\" (UniqueName: \"kubernetes.io/projected/9eedf2a6-9683-46f6-afbc-464ac0e312bc-kube-api-access-gv4z8\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.563815 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.563817 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eedf2a6-9683-46f6-afbc-464ac0e312bc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.563992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.563841 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eedf2a6-9683-46f6-afbc-464ac0e312bc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.563992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.563866 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eedf2a6-9683-46f6-afbc-464ac0e312bc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.563992 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.563886 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eedf2a6-9683-46f6-afbc-464ac0e312bc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.564137 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.564009 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eedf2a6-9683-46f6-afbc-464ac0e312bc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.564287 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.564264 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eedf2a6-9683-46f6-afbc-464ac0e312bc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.564362 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.564290 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eedf2a6-9683-46f6-afbc-464ac0e312bc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.564398 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.564356 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eedf2a6-9683-46f6-afbc-464ac0e312bc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.566166 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.566144 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eedf2a6-9683-46f6-afbc-464ac0e312bc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.566330 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.566284 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eedf2a6-9683-46f6-afbc-464ac0e312bc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.571627 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.571606 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv4z8\" (UniqueName: \"kubernetes.io/projected/9eedf2a6-9683-46f6-afbc-464ac0e312bc-kube-api-access-gv4z8\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ppckr\" (UID: \"9eedf2a6-9683-46f6-afbc-464ac0e312bc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.650886 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.650821 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:00.922239 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:00.922164 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr"] Sep 11 18:59:00.925032 ip-10-0-143-253 kubenswrapper[2556]: W0911 18:59:00.925006 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eedf2a6_9683_46f6_afbc_464ac0e312bc.slice/crio-752e9e9af5058faae7ada27c693b2926943dc1f7726fbd4de09e6462f739b85e WatchSource:0}: Error finding container 752e9e9af5058faae7ada27c693b2926943dc1f7726fbd4de09e6462f739b85e: Status 404 returned error can't find the container with id 752e9e9af5058faae7ada27c693b2926943dc1f7726fbd4de09e6462f739b85e Sep 11 18:59:01.130486 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:01.130449 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" event={"ID":"9eedf2a6-9683-46f6-afbc-464ac0e312bc","Type":"ContainerStarted","Data":"752e9e9af5058faae7ada27c693b2926943dc1f7726fbd4de09e6462f739b85e"} Sep 11 18:59:01.553527 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:01.553491 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:59:01.553924 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:01.553568 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:59:01.566179 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:01.566155 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:59:02.136707 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:02.136665 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" event={"ID":"9eedf2a6-9683-46f6-afbc-464ac0e312bc","Type":"ContainerStarted","Data":"09a81a13e55f702737c0d5c8666269d9abb61fe6c103fe48bfc0279a82821915"} Sep 11 18:59:02.147929 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:02.147904 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs" Sep 11 18:59:02.155338 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:02.155289 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" podStartSLOduration=1.9166368029999998 podStartE2EDuration="2.15527466s" totalImagesPullingTime="238.637857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:59:02.154619526 +0000 UTC m=+647.608425492" watchObservedRunningTime="2026-09-11 18:59:02.15527466 +0000 UTC m=+647.609080615" Sep 11 18:59:04.081900 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:04.081873 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:59:10.651927 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:10.651887 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:10.652346 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:10.651971 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:10.664130 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:10.664101 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:11.194550 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:11.194507 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ppckr" Sep 11 18:59:11.633449 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:11.633410 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 18:59:18.877101 ip-10-0-143-253 kubenswrapper[2556]: I0911 18:59:18.877068 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:00.144581 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:00.144536 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819220-bpgbc"] Sep 11 19:00:00.148153 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:00.148130 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819220-bpgbc" Sep 11 19:00:00.150336 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:00.150315 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\"" Sep 11 19:00:00.153529 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:00.153508 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819220-bpgbc"] Sep 11 19:00:00.159490 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:00.159470 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819220-bpgbc" Sep 11 19:00:00.427155 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:00.427133 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819220-bpgbc"] Sep 11 19:00:00.429789 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:00:00.429756 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e694c7b_4b7c_45fa_9ca9_eca594a7cb5e.slice/crio-69957173341e7b69cb07ca97fffafffa94ecc88bae882780ac716357dc6fb823 WatchSource:0}: Error finding container 69957173341e7b69cb07ca97fffafffa94ecc88bae882780ac716357dc6fb823: Status 404 returned error can't find the container with id 69957173341e7b69cb07ca97fffafffa94ecc88bae882780ac716357dc6fb823 Sep 11 19:00:01.409934 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:01.409903 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819220-bpgbc" event={"ID":"5e694c7b-4b7c-45fa-9ca9-eca594a7cb5e","Type":"ContainerStarted","Data":"69957173341e7b69cb07ca97fffafffa94ecc88bae882780ac716357dc6fb823"} Sep 11 19:00:01.887351 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:01.887318 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:04.428154 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:04.428116 2556 generic.go:356] "Generic (PLEG): container finished" podID="5e694c7b-4b7c-45fa-9ca9-eca594a7cb5e" containerID="a113ed6e6181cd4f1b1c2460702a3ed19ce178da4c74307d213a569e0c569bf8" exitCode=0 Sep 11 19:00:04.428516 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:04.428211 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819220-bpgbc" event={"ID":"5e694c7b-4b7c-45fa-9ca9-eca594a7cb5e","Type":"ContainerDied","Data":"a113ed6e6181cd4f1b1c2460702a3ed19ce178da4c74307d213a569e0c569bf8"} Sep 11 19:00:05.560995 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:05.560973 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819220-bpgbc" Sep 11 19:00:05.743258 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:05.743180 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:06.438400 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:06.438358 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819220-bpgbc" event={"ID":"5e694c7b-4b7c-45fa-9ca9-eca594a7cb5e","Type":"ContainerDied","Data":"69957173341e7b69cb07ca97fffafffa94ecc88bae882780ac716357dc6fb823"} Sep 11 19:00:06.438400 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:06.438395 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819220-bpgbc" Sep 11 19:00:06.438634 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:06.438423 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69957173341e7b69cb07ca97fffafffa94ecc88bae882780ac716357dc6fb823" Sep 11 19:00:20.398769 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:20.398666 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:20.497563 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:20.497508 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:21.524755 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:21.524725 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:21.641275 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:21.641248 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:22.969701 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:22.969661 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:37.064399 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:37.064359 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:37.128733 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:37.128695 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:37.200515 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:37.200481 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:37.403486 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:37.403397 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:37.501603 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:37.501569 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:37.576450 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:37.576416 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:37.654403 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:37.654310 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:38.045254 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:38.045222 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:38.436204 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:38.436122 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:38.533158 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:38.533118 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:38.874367 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:38.874335 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:40.373134 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:40.373098 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:40.473098 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:40.473061 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:40.578622 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:40.578594 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:40.906438 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:40.906402 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:41.007792 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:41.007751 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:41.107507 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:41.107477 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:42.177523 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:42.177488 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:42.276983 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:42.276949 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:42.502805 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:42.502713 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:42.800591 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:42.800559 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:42.905411 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:42.905379 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:42.935824 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:42.935789 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9"] Sep 11 19:00:42.936185 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:42.936172 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e694c7b-4b7c-45fa-9ca9-eca594a7cb5e" containerName="cleanup" Sep 11 19:00:42.936239 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:42.936188 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e694c7b-4b7c-45fa-9ca9-eca594a7cb5e" containerName="cleanup" Sep 11 19:00:42.936275 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:42.936260 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e694c7b-4b7c-45fa-9ca9-eca594a7cb5e" containerName="cleanup" Sep 11 19:00:42.939477 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:42.939461 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" Sep 11 19:00:42.941788 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:42.941760 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-c1aebc-dockercfg-rs7wf\"" Sep 11 19:00:42.941788 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:42.941774 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c1aebc\"" Sep 11 19:00:42.950001 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:42.949976 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9"] Sep 11 19:00:42.978110 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:42.978083 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv"] Sep 11 19:00:42.981652 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:42.981637 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" Sep 11 19:00:42.990069 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:42.990047 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv"] Sep 11 19:00:43.082914 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.082830 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps46k\" (UniqueName: \"kubernetes.io/projected/3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55-kube-api-access-ps46k\") pod \"payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9\" (UID: \"3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" Sep 11 19:00:43.082914 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.082872 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtw64\" (UniqueName: \"kubernetes.io/projected/b6a333da-fe91-4593-b69e-f49b76efba30-kube-api-access-dtw64\") pod \"payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv\" (UID: \"b6a333da-fe91-4593-b69e-f49b76efba30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" Sep 11 19:00:43.083138 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.082952 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9\" (UID: \"3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" Sep 11 19:00:43.083138 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.083087 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a333da-fe91-4593-b69e-f49b76efba30-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv\" (UID: \"b6a333da-fe91-4593-b69e-f49b76efba30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" Sep 11 19:00:43.184430 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.184398 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtw64\" (UniqueName: \"kubernetes.io/projected/b6a333da-fe91-4593-b69e-f49b76efba30-kube-api-access-dtw64\") pod \"payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv\" (UID: \"b6a333da-fe91-4593-b69e-f49b76efba30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" Sep 11 19:00:43.184876 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.184446 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9\" (UID: \"3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" Sep 11 19:00:43.184876 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.184502 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a333da-fe91-4593-b69e-f49b76efba30-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv\" (UID: \"b6a333da-fe91-4593-b69e-f49b76efba30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" Sep 11 19:00:43.184876 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.184530 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps46k\" (UniqueName: \"kubernetes.io/projected/3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55-kube-api-access-ps46k\") pod \"payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9\" (UID: \"3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" Sep 11 19:00:43.185195 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.185173 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9\" (UID: \"3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" Sep 11 19:00:43.185244 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.185225 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a333da-fe91-4593-b69e-f49b76efba30-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv\" (UID: \"b6a333da-fe91-4593-b69e-f49b76efba30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" Sep 11 19:00:43.192323 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.192292 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps46k\" (UniqueName: \"kubernetes.io/projected/3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55-kube-api-access-ps46k\") pod \"payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9\" (UID: \"3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" Sep 11 19:00:43.192423 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.192331 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtw64\" (UniqueName: \"kubernetes.io/projected/b6a333da-fe91-4593-b69e-f49b76efba30-kube-api-access-dtw64\") pod \"payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv\" (UID: \"b6a333da-fe91-4593-b69e-f49b76efba30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" Sep 11 19:00:43.250345 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.250313 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" Sep 11 19:00:43.291877 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.291838 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" Sep 11 19:00:43.453150 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.453116 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:43.524881 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.524802 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:43.526347 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.526304 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9"] Sep 11 19:00:43.528941 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:00:43.528915 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9e6360_c6e0_469d_bbbb_e8dd0cf7fe55.slice/crio-00906e80a5722be4c5f0321e54927f38d5432a3b3ea3a10a7dc756ca9233d999 WatchSource:0}: Error finding container 00906e80a5722be4c5f0321e54927f38d5432a3b3ea3a10a7dc756ca9233d999: Status 404 returned error can't find the container with id 00906e80a5722be4c5f0321e54927f38d5432a3b3ea3a10a7dc756ca9233d999 Sep 11 19:00:43.572654 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.572628 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv"] Sep 11 19:00:43.575356 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:00:43.575325 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a333da_fe91_4593_b69e_f49b76efba30.slice/crio-caac8f512924b0e390dd25e88b7cc04a086b8f7c37eef3258b1a3e3a261474f6 WatchSource:0}: Error finding container caac8f512924b0e390dd25e88b7cc04a086b8f7c37eef3258b1a3e3a261474f6: Status 404 returned error can't find the container with id caac8f512924b0e390dd25e88b7cc04a086b8f7c37eef3258b1a3e3a261474f6 Sep 11 19:00:43.630210 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.630182 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" event={"ID":"b6a333da-fe91-4593-b69e-f49b76efba30","Type":"ContainerStarted","Data":"fc88a4071bb59abcd9490331390b0c7a1976d5e7ab293d2cfedeef6be265a5fe"} Sep 11 19:00:43.630334 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.630233 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" event={"ID":"b6a333da-fe91-4593-b69e-f49b76efba30","Type":"ContainerStarted","Data":"caac8f512924b0e390dd25e88b7cc04a086b8f7c37eef3258b1a3e3a261474f6"} Sep 11 19:00:43.630334 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.630270 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" Sep 11 19:00:43.632130 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.632105 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" event={"ID":"3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55","Type":"ContainerStarted","Data":"1789bdd064f3fd2dc2ce944aa5b8682f1708eca1b0ccf31771d82e40a95e8169"} Sep 11 19:00:43.632237 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.632152 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" event={"ID":"3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55","Type":"ContainerStarted","Data":"00906e80a5722be4c5f0321e54927f38d5432a3b3ea3a10a7dc756ca9233d999"} Sep 11 19:00:43.632237 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.632219 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" Sep 11 19:00:43.654224 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.654175 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" podStartSLOduration=1.654159345 podStartE2EDuration="1.654159345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:43.652306701 +0000 UTC m=+749.106112677" watchObservedRunningTime="2026-09-11 19:00:43.654159345 +0000 UTC m=+749.107965299" Sep 11 19:00:43.670617 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.670560 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" podStartSLOduration=1.6705260800000001 podStartE2EDuration="1.67052608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:43.670151508 +0000 UTC m=+749.123957490" watchObservedRunningTime="2026-09-11 19:00:43.67052608 +0000 UTC m=+749.124332018" Sep 11 19:00:43.692377 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:43.692335 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:44.074673 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.074641 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:44.177032 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.176995 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:44.188234 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.188203 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f"] Sep 11 19:00:44.191888 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.191868 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" Sep 11 19:00:44.193774 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.193753 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-62499f\"" Sep 11 19:00:44.205948 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.205920 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f"] Sep 11 19:00:44.269879 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.269837 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl"] Sep 11 19:00:44.273770 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.273752 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" Sep 11 19:00:44.275919 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.275889 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-62499f-dockercfg-xspj9\"" Sep 11 19:00:44.281462 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.281440 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl"] Sep 11 19:00:44.295802 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.294960 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d574dd-4362-4e57-8dc0-b483d339f838-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f\" (UID: \"a8d574dd-4362-4e57-8dc0-b483d339f838\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" Sep 11 19:00:44.295802 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.295002 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f97nl\" (UniqueName: \"kubernetes.io/projected/a8d574dd-4362-4e57-8dc0-b483d339f838-kube-api-access-f97nl\") pod \"payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f\" (UID: \"a8d574dd-4362-4e57-8dc0-b483d339f838\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" Sep 11 19:00:44.395522 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.395441 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4m7r\" (UniqueName: \"kubernetes.io/projected/bc6eaa7b-901d-4bf0-86ff-672169131846-kube-api-access-q4m7r\") pod \"payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl\" (UID: \"bc6eaa7b-901d-4bf0-86ff-672169131846\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" Sep 11 19:00:44.395522 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.395480 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d574dd-4362-4e57-8dc0-b483d339f838-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f\" (UID: \"a8d574dd-4362-4e57-8dc0-b483d339f838\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" Sep 11 19:00:44.395522 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.395504 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f97nl\" (UniqueName: \"kubernetes.io/projected/a8d574dd-4362-4e57-8dc0-b483d339f838-kube-api-access-f97nl\") pod \"payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f\" (UID: \"a8d574dd-4362-4e57-8dc0-b483d339f838\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" Sep 11 19:00:44.395810 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.395611 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6eaa7b-901d-4bf0-86ff-672169131846-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl\" (UID: \"bc6eaa7b-901d-4bf0-86ff-672169131846\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" Sep 11 19:00:44.396234 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.396215 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d574dd-4362-4e57-8dc0-b483d339f838-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f\" (UID: \"a8d574dd-4362-4e57-8dc0-b483d339f838\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" Sep 11 19:00:44.402790 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.402767 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f97nl\" (UniqueName: \"kubernetes.io/projected/a8d574dd-4362-4e57-8dc0-b483d339f838-kube-api-access-f97nl\") pod \"payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f\" (UID: \"a8d574dd-4362-4e57-8dc0-b483d339f838\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" Sep 11 19:00:44.409735 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.409710 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:44.496803 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.496764 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6eaa7b-901d-4bf0-86ff-672169131846-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl\" (UID: \"bc6eaa7b-901d-4bf0-86ff-672169131846\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" Sep 11 19:00:44.496971 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.496879 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4m7r\" (UniqueName: \"kubernetes.io/projected/bc6eaa7b-901d-4bf0-86ff-672169131846-kube-api-access-q4m7r\") pod \"payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl\" (UID: \"bc6eaa7b-901d-4bf0-86ff-672169131846\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" Sep 11 19:00:44.497561 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.497527 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6eaa7b-901d-4bf0-86ff-672169131846-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl\" (UID: \"bc6eaa7b-901d-4bf0-86ff-672169131846\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" Sep 11 19:00:44.507428 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.507401 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4m7r\" (UniqueName: \"kubernetes.io/projected/bc6eaa7b-901d-4bf0-86ff-672169131846-kube-api-access-q4m7r\") pod \"payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl\" (UID: \"bc6eaa7b-901d-4bf0-86ff-672169131846\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" Sep 11 19:00:44.513197 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.513174 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" Sep 11 19:00:44.571929 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.571896 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:44.589479 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.588665 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" Sep 11 19:00:44.680384 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.680352 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:44.761276 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.761238 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:44.819490 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.819462 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f"] Sep 11 19:00:44.820834 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:00:44.820807 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d574dd_4362_4e57_8dc0_b483d339f838.slice/crio-81a7c46448a42c05cdb1606155d6946ad2062bb0f73984058cd06e06b8ab16c0 WatchSource:0}: Error finding container 81a7c46448a42c05cdb1606155d6946ad2062bb0f73984058cd06e06b8ab16c0: Status 404 returned error can't find the container with id 81a7c46448a42c05cdb1606155d6946ad2062bb0f73984058cd06e06b8ab16c0 Sep 11 19:00:44.888162 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:44.887520 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl"] Sep 11 19:00:44.890165 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:00:44.890140 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6eaa7b_901d_4bf0_86ff_672169131846.slice/crio-3480538f17f680f84c494c2cd222acc8975837d37d5cf133781b45186a9fd27d WatchSource:0}: Error finding container 3480538f17f680f84c494c2cd222acc8975837d37d5cf133781b45186a9fd27d: Status 404 returned error can't find the container with id 3480538f17f680f84c494c2cd222acc8975837d37d5cf133781b45186a9fd27d Sep 11 19:00:45.160483 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.160396 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:45.257057 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.257006 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:45.337141 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.337109 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:45.342125 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.342099 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp"] Sep 11 19:00:45.345638 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.345623 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" Sep 11 19:00:45.347678 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.347657 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c6c610-dockercfg-jh2jc\"" Sep 11 19:00:45.347875 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.347663 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c6c610\"" Sep 11 19:00:45.357131 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.357109 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp"] Sep 11 19:00:45.505250 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.505142 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br4g7\" (UniqueName: \"kubernetes.io/projected/697928b0-6a79-4a45-9d7a-a84ed8941237-kube-api-access-br4g7\") pod \"payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp\" (UID: \"697928b0-6a79-4a45-9d7a-a84ed8941237\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" Sep 11 19:00:45.505414 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.505295 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/697928b0-6a79-4a45-9d7a-a84ed8941237-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp\" (UID: \"697928b0-6a79-4a45-9d7a-a84ed8941237\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" Sep 11 19:00:45.602177 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.602140 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:45.606065 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.606041 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br4g7\" (UniqueName: \"kubernetes.io/projected/697928b0-6a79-4a45-9d7a-a84ed8941237-kube-api-access-br4g7\") pod \"payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp\" (UID: \"697928b0-6a79-4a45-9d7a-a84ed8941237\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" Sep 11 19:00:45.606148 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.606114 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/697928b0-6a79-4a45-9d7a-a84ed8941237-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp\" (UID: \"697928b0-6a79-4a45-9d7a-a84ed8941237\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" Sep 11 19:00:45.606835 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.606815 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/697928b0-6a79-4a45-9d7a-a84ed8941237-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp\" (UID: \"697928b0-6a79-4a45-9d7a-a84ed8941237\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" Sep 11 19:00:45.614014 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.613992 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br4g7\" (UniqueName: \"kubernetes.io/projected/697928b0-6a79-4a45-9d7a-a84ed8941237-kube-api-access-br4g7\") pod \"payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp\" (UID: \"697928b0-6a79-4a45-9d7a-a84ed8941237\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" Sep 11 19:00:45.647409 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.647367 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" event={"ID":"a8d574dd-4362-4e57-8dc0-b483d339f838","Type":"ContainerStarted","Data":"47e5d81353c8b3191583fdc01420acb7ace059941d0345194c870da81d1704ae"} Sep 11 19:00:45.647614 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.647522 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" event={"ID":"a8d574dd-4362-4e57-8dc0-b483d339f838","Type":"ContainerStarted","Data":"81a7c46448a42c05cdb1606155d6946ad2062bb0f73984058cd06e06b8ab16c0"} Sep 11 19:00:45.647614 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.647575 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" Sep 11 19:00:45.649928 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.649897 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" event={"ID":"bc6eaa7b-901d-4bf0-86ff-672169131846","Type":"ContainerStarted","Data":"f667eaf4bddb401042ed573021fb4e4285fbf6c72251a10299ec39606deb843d"} Sep 11 19:00:45.650028 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.649949 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" event={"ID":"bc6eaa7b-901d-4bf0-86ff-672169131846","Type":"ContainerStarted","Data":"3480538f17f680f84c494c2cd222acc8975837d37d5cf133781b45186a9fd27d"} Sep 11 19:00:45.650028 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.650003 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" Sep 11 19:00:45.660300 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.660273 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" Sep 11 19:00:45.700400 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.700333 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" podStartSLOduration=1.700312058 podStartE2EDuration="1.700312058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:45.679249901 +0000 UTC m=+751.133055873" watchObservedRunningTime="2026-09-11 19:00:45.700312058 +0000 UTC m=+751.154118016" Sep 11 19:00:45.700898 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.700856 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" podStartSLOduration=1.700842106 podStartE2EDuration="1.700842106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:45.699565396 +0000 UTC m=+751.153371364" watchObservedRunningTime="2026-09-11 19:00:45.700842106 +0000 UTC m=+751.154648060" Sep 11 19:00:45.940238 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:45.940208 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp"] Sep 11 19:00:45.942493 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:00:45.942464 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697928b0_6a79_4a45_9d7a_a84ed8941237.slice/crio-f222c5721eed0833fa84a4be9906ba23758224add0f1889a6da85a38b5b87bde WatchSource:0}: Error finding container f222c5721eed0833fa84a4be9906ba23758224add0f1889a6da85a38b5b87bde: Status 404 returned error can't find the container with id f222c5721eed0833fa84a4be9906ba23758224add0f1889a6da85a38b5b87bde Sep 11 19:00:46.463370 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:46.463324 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:46.567177 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:46.567138 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:46.658312 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:46.658214 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" event={"ID":"697928b0-6a79-4a45-9d7a-a84ed8941237","Type":"ContainerStarted","Data":"be50a82f55fbe8b5600b410c66eb384335a409c419413768de0d8b0495b426f3"} Sep 11 19:00:46.658312 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:46.658273 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" event={"ID":"697928b0-6a79-4a45-9d7a-a84ed8941237","Type":"ContainerStarted","Data":"f222c5721eed0833fa84a4be9906ba23758224add0f1889a6da85a38b5b87bde"} Sep 11 19:00:46.658600 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:46.658471 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" Sep 11 19:00:46.667850 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:46.667821 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:46.679274 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:46.679220 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" podStartSLOduration=1.6792000630000001 podStartE2EDuration="1.679200063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:46.67658154 +0000 UTC m=+752.130387521" watchObservedRunningTime="2026-09-11 19:00:46.679200063 +0000 UTC m=+752.133006022" Sep 11 19:00:46.798827 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:46.798780 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:47.123488 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:47.123392 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:47.451022 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:47.448017 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:47.560933 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:47.560901 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:48.318794 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:48.318762 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:48.402477 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:48.402443 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:48.528273 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:48.528234 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:49.702691 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:49.702653 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:49.801656 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:49.801608 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:49.905886 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:49.905856 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:50.007019 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:50.006937 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:50.953715 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:50.953682 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:52.333140 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:52.333099 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:53.403793 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:53.403760 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:53.467885 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:53.467850 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:53.613126 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:53.613081 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:54.216713 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.216682 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:54.317526 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.317491 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:54.436756 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.436716 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb"] Sep 11 19:00:54.440704 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.440680 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" Sep 11 19:00:54.443041 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.443021 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-aff2b8e2-dockercfg-nnm2x\"" Sep 11 19:00:54.443158 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.443046 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-aff2b8e2\"" Sep 11 19:00:54.453444 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.452166 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb"] Sep 11 19:00:54.521474 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.521395 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh"] Sep 11 19:00:54.525015 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.524997 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" Sep 11 19:00:54.535742 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.535716 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh"] Sep 11 19:00:54.590511 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.590470 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd822\" (UniqueName: \"kubernetes.io/projected/e60f0eb1-1fde-4b39-980b-d347bef49899-kube-api-access-qd822\") pod \"payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb\" (UID: \"e60f0eb1-1fde-4b39-980b-d347bef49899\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" Sep 11 19:00:54.590511 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.590516 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60f0eb1-1fde-4b39-980b-d347bef49899-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb\" (UID: \"e60f0eb1-1fde-4b39-980b-d347bef49899\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" Sep 11 19:00:54.643092 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.643063 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" Sep 11 19:00:54.643220 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.643135 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" Sep 11 19:00:54.691236 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.691202 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmdlc\" (UniqueName: \"kubernetes.io/projected/44930a7b-15c0-4f25-be75-36251e703557-kube-api-access-wmdlc\") pod \"payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh\" (UID: \"44930a7b-15c0-4f25-be75-36251e703557\") " pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" Sep 11 19:00:54.691413 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.691295 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44930a7b-15c0-4f25-be75-36251e703557-plugins-config-volume\") pod \"payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh\" (UID: \"44930a7b-15c0-4f25-be75-36251e703557\") " pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" Sep 11 19:00:54.691413 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.691344 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd822\" (UniqueName: \"kubernetes.io/projected/e60f0eb1-1fde-4b39-980b-d347bef49899-kube-api-access-qd822\") pod \"payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb\" (UID: \"e60f0eb1-1fde-4b39-980b-d347bef49899\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" Sep 11 19:00:54.691530 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.691430 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60f0eb1-1fde-4b39-980b-d347bef49899-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb\" (UID: \"e60f0eb1-1fde-4b39-980b-d347bef49899\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" Sep 11 19:00:54.692256 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.692233 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60f0eb1-1fde-4b39-980b-d347bef49899-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb\" (UID: \"e60f0eb1-1fde-4b39-980b-d347bef49899\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" Sep 11 19:00:54.701095 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.701062 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd822\" (UniqueName: \"kubernetes.io/projected/e60f0eb1-1fde-4b39-980b-d347bef49899-kube-api-access-qd822\") pod \"payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb\" (UID: \"e60f0eb1-1fde-4b39-980b-d347bef49899\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" Sep 11 19:00:54.764727 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.764682 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" Sep 11 19:00:54.793047 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.793012 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44930a7b-15c0-4f25-be75-36251e703557-plugins-config-volume\") pod \"payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh\" (UID: \"44930a7b-15c0-4f25-be75-36251e703557\") " pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" Sep 11 19:00:54.793174 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.793161 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmdlc\" (UniqueName: \"kubernetes.io/projected/44930a7b-15c0-4f25-be75-36251e703557-kube-api-access-wmdlc\") pod \"payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh\" (UID: \"44930a7b-15c0-4f25-be75-36251e703557\") " pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" Sep 11 19:00:54.793886 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.793858 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44930a7b-15c0-4f25-be75-36251e703557-plugins-config-volume\") pod \"payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh\" (UID: \"44930a7b-15c0-4f25-be75-36251e703557\") " pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" Sep 11 19:00:54.805171 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.805101 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmdlc\" (UniqueName: \"kubernetes.io/projected/44930a7b-15c0-4f25-be75-36251e703557-kube-api-access-wmdlc\") pod \"payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh\" (UID: \"44930a7b-15c0-4f25-be75-36251e703557\") " pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" Sep 11 19:00:54.821496 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.821470 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:54.837168 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.837146 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" Sep 11 19:00:54.923647 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:54.923614 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:55.022089 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:55.022058 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:55.052956 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:55.052928 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb"] Sep 11 19:00:55.057475 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:00:55.057450 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60f0eb1_1fde_4b39_980b_d347bef49899.slice/crio-74c57eb15b604bca7f08f67de60ae855eecf1df0dfeacb1d38331513f6d9094f WatchSource:0}: Error finding container 74c57eb15b604bca7f08f67de60ae855eecf1df0dfeacb1d38331513f6d9094f: Status 404 returned error can't find the container with id 74c57eb15b604bca7f08f67de60ae855eecf1df0dfeacb1d38331513f6d9094f Sep 11 19:00:55.126925 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:55.126866 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh"] Sep 11 19:00:55.128911 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:00:55.128886 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44930a7b_15c0_4f25_be75_36251e703557.slice/crio-360c44e76e660a62c65dc7769d769c1ded539ce93f1c7361c1c1e299761ca1b5 WatchSource:0}: Error finding container 360c44e76e660a62c65dc7769d769c1ded539ce93f1c7361c1c1e299761ca1b5: Status 404 returned error can't find the container with id 360c44e76e660a62c65dc7769d769c1ded539ce93f1c7361c1c1e299761ca1b5 Sep 11 19:00:55.508169 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:55.508079 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:55.656929 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:55.656895 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:55.704990 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:55.704956 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" event={"ID":"e60f0eb1-1fde-4b39-980b-d347bef49899","Type":"ContainerStarted","Data":"c17bb8ecad017cf7d4046402bc2a5073296173ce51f5d71a08acaa14d55606f2"} Sep 11 19:00:55.705146 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:55.705035 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" event={"ID":"e60f0eb1-1fde-4b39-980b-d347bef49899","Type":"ContainerStarted","Data":"74c57eb15b604bca7f08f67de60ae855eecf1df0dfeacb1d38331513f6d9094f"} Sep 11 19:00:55.705146 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:55.705070 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" Sep 11 19:00:55.706887 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:55.706858 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" event={"ID":"44930a7b-15c0-4f25-be75-36251e703557","Type":"ContainerStarted","Data":"af1d2c071d6919084ea53fe4f3b3cb8bf60dff43035e77b727b2a6e427a66006"} Sep 11 19:00:55.706998 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:55.706908 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" event={"ID":"44930a7b-15c0-4f25-be75-36251e703557","Type":"ContainerStarted","Data":"360c44e76e660a62c65dc7769d769c1ded539ce93f1c7361c1c1e299761ca1b5"} Sep 11 19:00:55.706998 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:55.706936 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" Sep 11 19:00:55.732577 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:55.732516 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" podStartSLOduration=1.732480216 podStartE2EDuration="1.732480216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:55.731403177 +0000 UTC m=+761.185209161" watchObservedRunningTime="2026-09-11 19:00:55.732480216 +0000 UTC m=+761.186286195" Sep 11 19:00:55.765683 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:55.765577 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" podStartSLOduration=1.7655537510000001 podStartE2EDuration="1.765553751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:55.763606626 +0000 UTC m=+761.217412582" watchObservedRunningTime="2026-09-11 19:00:55.765553751 +0000 UTC m=+761.219359702" Sep 11 19:00:55.771707 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:55.771677 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:55.898427 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:55.898393 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:55.999151 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:55.999110 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:56.108120 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.108084 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:56.191303 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.191272 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:56.272689 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.272656 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:56.303981 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.303903 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp"] Sep 11 19:00:56.308391 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.308367 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" Sep 11 19:00:56.312437 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.311621 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-89e163-dockercfg-mpb7t\"" Sep 11 19:00:56.312437 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.311873 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-89e163\"" Sep 11 19:00:56.324105 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.324080 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp"] Sep 11 19:00:56.370391 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.370299 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:56.408036 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.407998 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6djc\" (UniqueName: \"kubernetes.io/projected/edb67128-4a54-48bd-a7cb-90efacade95e-kube-api-access-w6djc\") pod \"payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp\" (UID: \"edb67128-4a54-48bd-a7cb-90efacade95e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" Sep 11 19:00:56.408212 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.408060 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edb67128-4a54-48bd-a7cb-90efacade95e-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp\" (UID: \"edb67128-4a54-48bd-a7cb-90efacade95e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" Sep 11 19:00:56.473918 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.473883 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:56.509111 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.509080 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6djc\" (UniqueName: \"kubernetes.io/projected/edb67128-4a54-48bd-a7cb-90efacade95e-kube-api-access-w6djc\") pod \"payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp\" (UID: \"edb67128-4a54-48bd-a7cb-90efacade95e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" Sep 11 19:00:56.509502 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.509131 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edb67128-4a54-48bd-a7cb-90efacade95e-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp\" (UID: \"edb67128-4a54-48bd-a7cb-90efacade95e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" Sep 11 19:00:56.509883 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.509863 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edb67128-4a54-48bd-a7cb-90efacade95e-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp\" (UID: \"edb67128-4a54-48bd-a7cb-90efacade95e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" Sep 11 19:00:56.517262 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.517238 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6djc\" (UniqueName: \"kubernetes.io/projected/edb67128-4a54-48bd-a7cb-90efacade95e-kube-api-access-w6djc\") pod \"payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp\" (UID: \"edb67128-4a54-48bd-a7cb-90efacade95e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" Sep 11 19:00:56.625161 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.625072 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" Sep 11 19:00:56.661063 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.661027 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" Sep 11 19:00:56.661210 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.661128 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" Sep 11 19:00:56.720022 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.719989 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:56.904552 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:56.904511 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp"] Sep 11 19:00:56.907173 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:00:56.907138 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb67128_4a54_48bd_a7cb_90efacade95e.slice/crio-6f9560e835712d9fc82d610cae3305fa8a2bfdc6e0368538416d0050026c22dd WatchSource:0}: Error finding container 6f9560e835712d9fc82d610cae3305fa8a2bfdc6e0368538416d0050026c22dd: Status 404 returned error can't find the container with id 6f9560e835712d9fc82d610cae3305fa8a2bfdc6e0368538416d0050026c22dd Sep 11 19:00:57.326486 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:57.326443 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:57.435839 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:57.435806 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:57.530130 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:57.530094 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:57.667221 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:57.667141 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" Sep 11 19:00:57.722627 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:57.722589 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" event={"ID":"edb67128-4a54-48bd-a7cb-90efacade95e","Type":"ContainerStarted","Data":"3ad37b33572cb136de1cee3156bee96596aa6c3e75415e6881022d09b589f042"} Sep 11 19:00:57.722802 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:57.722658 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" event={"ID":"edb67128-4a54-48bd-a7cb-90efacade95e","Type":"ContainerStarted","Data":"6f9560e835712d9fc82d610cae3305fa8a2bfdc6e0368538416d0050026c22dd"} Sep 11 19:00:57.722802 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:57.722692 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" Sep 11 19:00:57.731978 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:57.731952 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:57.756198 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:57.756148 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" podStartSLOduration=1.75613127 podStartE2EDuration="1.75613127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:00:57.751974347 +0000 UTC m=+763.205780325" watchObservedRunningTime="2026-09-11 19:00:57.75613127 +0000 UTC m=+763.209937262" Sep 11 19:00:57.887524 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:57.887488 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:58.416690 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:58.416655 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:58.578709 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:58.578674 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:58.679361 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:58.679279 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:58.779695 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:58.779665 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:59.175997 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:59.175965 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:59.447911 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:59.447801 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:00:59.539511 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:00:59.539479 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:00.070468 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:00.070433 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:01.093011 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:01.092983 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:01.418883 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:01.418796 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:01.918862 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:01.918836 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:02.906751 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:02.906712 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:03.005469 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.005440 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:03.105283 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.105254 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:03.449332 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.449300 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:03.621470 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.621433 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:03.727461 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.727384 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:03.833225 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.833194 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc"] Sep 11 19:01:03.837047 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.837030 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" Sep 11 19:01:03.839134 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.839106 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-093f6c\"" Sep 11 19:01:03.839134 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.839124 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-093f6c-dockercfg-qfqkd\"" Sep 11 19:01:03.849785 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.849758 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc"] Sep 11 19:01:03.865888 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.865860 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6213b5-0cb2-41f8-8e6a-66ed53bb3230-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc\" (UID: \"da6213b5-0cb2-41f8-8e6a-66ed53bb3230\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" Sep 11 19:01:03.866000 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.865893 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff7xp\" (UniqueName: \"kubernetes.io/projected/da6213b5-0cb2-41f8-8e6a-66ed53bb3230-kube-api-access-ff7xp\") pod \"payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc\" (UID: \"da6213b5-0cb2-41f8-8e6a-66ed53bb3230\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" Sep 11 19:01:03.924039 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.923999 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp"] Sep 11 19:01:03.927724 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.927700 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" Sep 11 19:01:03.939729 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.939705 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp"] Sep 11 19:01:03.966691 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.966662 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6213b5-0cb2-41f8-8e6a-66ed53bb3230-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc\" (UID: \"da6213b5-0cb2-41f8-8e6a-66ed53bb3230\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" Sep 11 19:01:03.966818 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.966701 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff7xp\" (UniqueName: \"kubernetes.io/projected/da6213b5-0cb2-41f8-8e6a-66ed53bb3230-kube-api-access-ff7xp\") pod \"payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc\" (UID: \"da6213b5-0cb2-41f8-8e6a-66ed53bb3230\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" Sep 11 19:01:03.966818 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.966785 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a89864-c5e3-42fe-9b71-4f0be63be7c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp\" (UID: \"74a89864-c5e3-42fe-9b71-4f0be63be7c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" Sep 11 19:01:03.966926 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.966825 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99r7n\" (UniqueName: \"kubernetes.io/projected/74a89864-c5e3-42fe-9b71-4f0be63be7c9-kube-api-access-99r7n\") pod \"payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp\" (UID: \"74a89864-c5e3-42fe-9b71-4f0be63be7c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" Sep 11 19:01:03.967391 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.967373 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6213b5-0cb2-41f8-8e6a-66ed53bb3230-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc\" (UID: \"da6213b5-0cb2-41f8-8e6a-66ed53bb3230\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" Sep 11 19:01:03.980377 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:03.980324 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff7xp\" (UniqueName: \"kubernetes.io/projected/da6213b5-0cb2-41f8-8e6a-66ed53bb3230-kube-api-access-ff7xp\") pod \"payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc\" (UID: \"da6213b5-0cb2-41f8-8e6a-66ed53bb3230\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" Sep 11 19:01:04.067496 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.067460 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a89864-c5e3-42fe-9b71-4f0be63be7c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp\" (UID: \"74a89864-c5e3-42fe-9b71-4f0be63be7c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" Sep 11 19:01:04.067655 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.067502 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99r7n\" (UniqueName: \"kubernetes.io/projected/74a89864-c5e3-42fe-9b71-4f0be63be7c9-kube-api-access-99r7n\") pod \"payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp\" (UID: \"74a89864-c5e3-42fe-9b71-4f0be63be7c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" Sep 11 19:01:04.068129 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.068109 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a89864-c5e3-42fe-9b71-4f0be63be7c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp\" (UID: \"74a89864-c5e3-42fe-9b71-4f0be63be7c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" Sep 11 19:01:04.075168 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.075136 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99r7n\" (UniqueName: \"kubernetes.io/projected/74a89864-c5e3-42fe-9b71-4f0be63be7c9-kube-api-access-99r7n\") pod \"payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp\" (UID: \"74a89864-c5e3-42fe-9b71-4f0be63be7c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" Sep 11 19:01:04.149972 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.149946 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" Sep 11 19:01:04.239911 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.239847 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" Sep 11 19:01:04.294915 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.294867 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:04.421445 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.421412 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc"] Sep 11 19:01:04.424563 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:01:04.424522 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6213b5_0cb2_41f8_8e6a_66ed53bb3230.slice/crio-2da001c35197e639bf1aa5138735ff2c0488cbf3c97a67c54391140f4666d8a8 WatchSource:0}: Error finding container 2da001c35197e639bf1aa5138735ff2c0488cbf3c97a67c54391140f4666d8a8: Status 404 returned error can't find the container with id 2da001c35197e639bf1aa5138735ff2c0488cbf3c97a67c54391140f4666d8a8 Sep 11 19:01:04.521156 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.521131 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp"] Sep 11 19:01:04.522497 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:01:04.522471 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a89864_c5e3_42fe_9b71_4f0be63be7c9.slice/crio-a86c7d81ebf335f503e262cf33d24fd815c57c0fb31addf515ced32fbc7628c8 WatchSource:0}: Error finding container a86c7d81ebf335f503e262cf33d24fd815c57c0fb31addf515ced32fbc7628c8: Status 404 returned error can't find the container with id a86c7d81ebf335f503e262cf33d24fd815c57c0fb31addf515ced32fbc7628c8 Sep 11 19:01:04.763061 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.762964 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" event={"ID":"da6213b5-0cb2-41f8-8e6a-66ed53bb3230","Type":"ContainerStarted","Data":"60bb260a0451302ab662d8847169e999d60e6f82f9f8c9c3a9c3732512850289"} Sep 11 19:01:04.763061 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.763032 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" event={"ID":"da6213b5-0cb2-41f8-8e6a-66ed53bb3230","Type":"ContainerStarted","Data":"2da001c35197e639bf1aa5138735ff2c0488cbf3c97a67c54391140f4666d8a8"} Sep 11 19:01:04.763265 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.763217 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" Sep 11 19:01:04.764873 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.764842 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" event={"ID":"74a89864-c5e3-42fe-9b71-4f0be63be7c9","Type":"ContainerStarted","Data":"e5a557cf072c474293f35cef624e47e4bec31382a7376ebc96819e4d9c0c9a15"} Sep 11 19:01:04.765241 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.764889 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" Sep 11 19:01:04.765241 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.764912 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" event={"ID":"74a89864-c5e3-42fe-9b71-4f0be63be7c9","Type":"ContainerStarted","Data":"a86c7d81ebf335f503e262cf33d24fd815c57c0fb31addf515ced32fbc7628c8"} Sep 11 19:01:04.781755 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.781716 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" podStartSLOduration=1.781702844 podStartE2EDuration="1.781702844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:01:04.779705365 +0000 UTC m=+770.233511344" watchObservedRunningTime="2026-09-11 19:01:04.781702844 +0000 UTC m=+770.235508799" Sep 11 19:01:04.797767 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.797730 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" podStartSLOduration=1.797717878 podStartE2EDuration="1.797717878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:01:04.795201255 +0000 UTC m=+770.249007237" watchObservedRunningTime="2026-09-11 19:01:04.797717878 +0000 UTC m=+770.251523833" Sep 11 19:01:04.925483 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:04.925450 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:05.057524 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:05.057493 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:05.635435 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:05.635401 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:06.631660 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:06.631626 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:06.714729 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:06.714702 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" Sep 11 19:01:06.714876 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:06.714779 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" Sep 11 19:01:07.582041 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:07.582011 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:08.679162 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:08.679123 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:08.730734 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:08.730713 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" Sep 11 19:01:14.733470 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:14.733441 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:15.772191 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:15.772161 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" Sep 11 19:01:15.772618 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:15.772233 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" Sep 11 19:01:16.255528 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:16.255478 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:17.061943 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:17.061908 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:17.942139 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:17.942103 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:18.395997 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:18.395958 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:18.739151 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:18.739070 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:19.233813 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:19.233768 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:19.686995 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:19.686964 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:20.444264 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:20.444229 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd"] Sep 11 19:01:20.449259 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:20.449223 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd" Sep 11 19:01:20.449399 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:20.449324 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:20.452693 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:20.452674 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\"" Sep 11 19:01:20.456836 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:20.456812 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd"] Sep 11 19:01:20.525221 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:20.525191 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f22b59d-2dac-4285-a5c0-f70d5637a826-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd\" (UID: \"8f22b59d-2dac-4285-a5c0-f70d5637a826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd" Sep 11 19:01:20.625889 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:20.625862 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f22b59d-2dac-4285-a5c0-f70d5637a826-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd\" (UID: \"8f22b59d-2dac-4285-a5c0-f70d5637a826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd" Sep 11 19:01:20.626450 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:20.626433 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f22b59d-2dac-4285-a5c0-f70d5637a826-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd\" (UID: \"8f22b59d-2dac-4285-a5c0-f70d5637a826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd" Sep 11 19:01:20.762606 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:20.762506 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd" Sep 11 19:01:21.030422 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:21.030399 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd"] Sep 11 19:01:21.032370 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:01:21.032335 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f22b59d_2dac_4285_a5c0_f70d5637a826.slice/crio-cc0150e983e1519ed43829a75df0c0f460aed1f2c03b7b3f39abed2974a4107b WatchSource:0}: Error finding container cc0150e983e1519ed43829a75df0c0f460aed1f2c03b7b3f39abed2974a4107b: Status 404 returned error can't find the container with id cc0150e983e1519ed43829a75df0c0f460aed1f2c03b7b3f39abed2974a4107b Sep 11 19:01:21.286315 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:21.286287 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:21.864329 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:21.864295 2556 generic.go:356] "Generic (PLEG): container finished" podID="8f22b59d-2dac-4285-a5c0-f70d5637a826" containerID="cd6cbdfc4fa25fbac9944d4a94bdb0eeba226b1c29c453b94ad6df99384e07c9" exitCode=0 Sep 11 19:01:21.864748 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:21.864341 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd" event={"ID":"8f22b59d-2dac-4285-a5c0-f70d5637a826","Type":"ContainerDied","Data":"cd6cbdfc4fa25fbac9944d4a94bdb0eeba226b1c29c453b94ad6df99384e07c9"} Sep 11 19:01:21.864748 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:21.864382 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd" event={"ID":"8f22b59d-2dac-4285-a5c0-f70d5637a826","Type":"ContainerStarted","Data":"cc0150e983e1519ed43829a75df0c0f460aed1f2c03b7b3f39abed2974a4107b"} Sep 11 19:01:22.033568 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:22.033518 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:22.965481 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:22.965444 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:23.000802 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:23.000774 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd" Sep 11 19:01:23.149095 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:23.149011 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f22b59d-2dac-4285-a5c0-f70d5637a826-maas-api-service-ca\") pod \"8f22b59d-2dac-4285-a5c0-f70d5637a826\" (UID: \"8f22b59d-2dac-4285-a5c0-f70d5637a826\") " Sep 11 19:01:23.149412 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:23.149382 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f22b59d-2dac-4285-a5c0-f70d5637a826-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f22b59d-2dac-4285-a5c0-f70d5637a826" (UID: "8f22b59d-2dac-4285-a5c0-f70d5637a826"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:01:23.249860 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:23.249835 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f22b59d-2dac-4285-a5c0-f70d5637a826-maas-api-service-ca\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:01:23.681139 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:23.681108 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:23.875996 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:23.875962 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd" Sep 11 19:01:23.876156 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:23.876014 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd" event={"ID":"8f22b59d-2dac-4285-a5c0-f70d5637a826","Type":"ContainerDied","Data":"cc0150e983e1519ed43829a75df0c0f460aed1f2c03b7b3f39abed2974a4107b"} Sep 11 19:01:23.876156 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:23.876060 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc0150e983e1519ed43829a75df0c0f460aed1f2c03b7b3f39abed2974a4107b" Sep 11 19:01:24.561199 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:24.561161 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:25.272274 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:25.272238 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:25.968357 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:25.968326 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:26.519102 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:26.519071 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:26.872952 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:26.872917 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:30.769724 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:30.769691 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:31.366514 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:31.366485 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:32.087369 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.087330 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:32.123496 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.123466 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh"] Sep 11 19:01:32.123759 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.123731 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" podUID="44930a7b-15c0-4f25-be75-36251e703557" containerName="payload-processing" containerID="cri-o://af1d2c071d6919084ea53fe4f3b3cb8bf60dff43035e77b727b2a6e427a66006" gracePeriod=30 Sep 11 19:01:32.130618 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.130591 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb"] Sep 11 19:01:32.132481 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.132451 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" podUID="e60f0eb1-1fde-4b39-980b-d347bef49899" containerName="payload-pre-processing" containerID="cri-o://c17bb8ecad017cf7d4046402bc2a5073296173ce51f5d71a08acaa14d55606f2" gracePeriod=30 Sep 11 19:01:32.172264 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.172236 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:32.245597 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.245566 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:32.326022 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.325992 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd"] Sep 11 19:01:32.331259 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.331236 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aff2b8e2-mqgsd"] Sep 11 19:01:32.337024 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.336982 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:32.400106 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.400084 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" Sep 11 19:01:32.429582 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.429530 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" Sep 11 19:01:32.521905 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.521865 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:32.535255 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.535233 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60f0eb1-1fde-4b39-980b-d347bef49899-plugins-config-volume\") pod \"e60f0eb1-1fde-4b39-980b-d347bef49899\" (UID: \"e60f0eb1-1fde-4b39-980b-d347bef49899\") " Sep 11 19:01:32.535255 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.535259 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44930a7b-15c0-4f25-be75-36251e703557-plugins-config-volume\") pod \"44930a7b-15c0-4f25-be75-36251e703557\" (UID: \"44930a7b-15c0-4f25-be75-36251e703557\") " Sep 11 19:01:32.535464 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.535302 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qd822\" (UniqueName: \"kubernetes.io/projected/e60f0eb1-1fde-4b39-980b-d347bef49899-kube-api-access-qd822\") pod \"e60f0eb1-1fde-4b39-980b-d347bef49899\" (UID: \"e60f0eb1-1fde-4b39-980b-d347bef49899\") " Sep 11 19:01:32.535464 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.535331 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmdlc\" (UniqueName: \"kubernetes.io/projected/44930a7b-15c0-4f25-be75-36251e703557-kube-api-access-wmdlc\") pod \"44930a7b-15c0-4f25-be75-36251e703557\" (UID: \"44930a7b-15c0-4f25-be75-36251e703557\") " Sep 11 19:01:32.535701 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.535679 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60f0eb1-1fde-4b39-980b-d347bef49899-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e60f0eb1-1fde-4b39-980b-d347bef49899" (UID: "e60f0eb1-1fde-4b39-980b-d347bef49899"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:01:32.535701 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.535684 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44930a7b-15c0-4f25-be75-36251e703557-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44930a7b-15c0-4f25-be75-36251e703557" (UID: "44930a7b-15c0-4f25-be75-36251e703557"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:01:32.537500 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.537477 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60f0eb1-1fde-4b39-980b-d347bef49899-kube-api-access-qd822" (OuterVolumeSpecName: "kube-api-access-qd822") pod "e60f0eb1-1fde-4b39-980b-d347bef49899" (UID: "e60f0eb1-1fde-4b39-980b-d347bef49899"). InnerVolumeSpecName "kube-api-access-qd822". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:01:32.537616 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.537504 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44930a7b-15c0-4f25-be75-36251e703557-kube-api-access-wmdlc" (OuterVolumeSpecName: "kube-api-access-wmdlc") pod "44930a7b-15c0-4f25-be75-36251e703557" (UID: "44930a7b-15c0-4f25-be75-36251e703557"). InnerVolumeSpecName "kube-api-access-wmdlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:01:32.598717 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.598650 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:32.636510 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.636484 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd822\" (UniqueName: \"kubernetes.io/projected/e60f0eb1-1fde-4b39-980b-d347bef49899-kube-api-access-qd822\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:01:32.636510 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.636509 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmdlc\" (UniqueName: \"kubernetes.io/projected/44930a7b-15c0-4f25-be75-36251e703557-kube-api-access-wmdlc\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:01:32.636691 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.636524 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e60f0eb1-1fde-4b39-980b-d347bef49899-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:01:32.636691 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.636552 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44930a7b-15c0-4f25-be75-36251e703557-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:01:32.872941 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.872842 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:32.930489 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.930457 2556 generic.go:356] "Generic (PLEG): container finished" podID="e60f0eb1-1fde-4b39-980b-d347bef49899" containerID="c17bb8ecad017cf7d4046402bc2a5073296173ce51f5d71a08acaa14d55606f2" exitCode=0 Sep 11 19:01:32.930658 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.930496 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" event={"ID":"e60f0eb1-1fde-4b39-980b-d347bef49899","Type":"ContainerDied","Data":"c17bb8ecad017cf7d4046402bc2a5073296173ce51f5d71a08acaa14d55606f2"} Sep 11 19:01:32.930658 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.930505 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" Sep 11 19:01:32.930658 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.930594 2556 scope.go:117] "RemoveContainer" containerID="c17bb8ecad017cf7d4046402bc2a5073296173ce51f5d71a08acaa14d55606f2" Sep 11 19:01:32.930817 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.930579 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb" event={"ID":"e60f0eb1-1fde-4b39-980b-d347bef49899","Type":"ContainerDied","Data":"74c57eb15b604bca7f08f67de60ae855eecf1df0dfeacb1d38331513f6d9094f"} Sep 11 19:01:32.932044 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.932021 2556 generic.go:356] "Generic (PLEG): container finished" podID="44930a7b-15c0-4f25-be75-36251e703557" containerID="af1d2c071d6919084ea53fe4f3b3cb8bf60dff43035e77b727b2a6e427a66006" exitCode=0 Sep 11 19:01:32.932157 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.932061 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" Sep 11 19:01:32.932157 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.932122 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" event={"ID":"44930a7b-15c0-4f25-be75-36251e703557","Type":"ContainerDied","Data":"af1d2c071d6919084ea53fe4f3b3cb8bf60dff43035e77b727b2a6e427a66006"} Sep 11 19:01:32.932261 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.932160 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh" event={"ID":"44930a7b-15c0-4f25-be75-36251e703557","Type":"ContainerDied","Data":"360c44e76e660a62c65dc7769d769c1ded539ce93f1c7361c1c1e299761ca1b5"} Sep 11 19:01:32.941236 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.941216 2556 scope.go:117] "RemoveContainer" containerID="c17bb8ecad017cf7d4046402bc2a5073296173ce51f5d71a08acaa14d55606f2" Sep 11 19:01:32.941691 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:01:32.941568 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17bb8ecad017cf7d4046402bc2a5073296173ce51f5d71a08acaa14d55606f2\": container with ID starting with c17bb8ecad017cf7d4046402bc2a5073296173ce51f5d71a08acaa14d55606f2 not found: ID does not exist" containerID="c17bb8ecad017cf7d4046402bc2a5073296173ce51f5d71a08acaa14d55606f2" Sep 11 19:01:32.941691 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.941606 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17bb8ecad017cf7d4046402bc2a5073296173ce51f5d71a08acaa14d55606f2"} err="failed to get container status \"c17bb8ecad017cf7d4046402bc2a5073296173ce51f5d71a08acaa14d55606f2\": rpc error: code = NotFound desc = could not find container \"c17bb8ecad017cf7d4046402bc2a5073296173ce51f5d71a08acaa14d55606f2\": container with ID starting with c17bb8ecad017cf7d4046402bc2a5073296173ce51f5d71a08acaa14d55606f2 not found: ID does not exist" Sep 11 19:01:32.941691 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.941632 2556 scope.go:117] "RemoveContainer" containerID="af1d2c071d6919084ea53fe4f3b3cb8bf60dff43035e77b727b2a6e427a66006" Sep 11 19:01:32.950757 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.950742 2556 scope.go:117] "RemoveContainer" containerID="af1d2c071d6919084ea53fe4f3b3cb8bf60dff43035e77b727b2a6e427a66006" Sep 11 19:01:32.951040 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:01:32.951022 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1d2c071d6919084ea53fe4f3b3cb8bf60dff43035e77b727b2a6e427a66006\": container with ID starting with af1d2c071d6919084ea53fe4f3b3cb8bf60dff43035e77b727b2a6e427a66006 not found: ID does not exist" containerID="af1d2c071d6919084ea53fe4f3b3cb8bf60dff43035e77b727b2a6e427a66006" Sep 11 19:01:32.951098 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.951050 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1d2c071d6919084ea53fe4f3b3cb8bf60dff43035e77b727b2a6e427a66006"} err="failed to get container status \"af1d2c071d6919084ea53fe4f3b3cb8bf60dff43035e77b727b2a6e427a66006\": rpc error: code = NotFound desc = could not find container \"af1d2c071d6919084ea53fe4f3b3cb8bf60dff43035e77b727b2a6e427a66006\": container with ID starting with af1d2c071d6919084ea53fe4f3b3cb8bf60dff43035e77b727b2a6e427a66006 not found: ID does not exist" Sep 11 19:01:32.963295 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.963272 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb"] Sep 11 19:01:32.971438 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.971397 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:32.973374 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.973356 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aff2b8e2-869b4874-9ztfb"] Sep 11 19:01:32.986095 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.986076 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh"] Sep 11 19:01:32.998321 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:32.998302 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aff2b8e2-78cdcdd9ff-4pzwh"] Sep 11 19:01:33.053155 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:33.053126 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:33.171961 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:33.171925 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:33.172618 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:33.172599 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44930a7b-15c0-4f25-be75-36251e703557" path="/var/lib/kubelet/pods/44930a7b-15c0-4f25-be75-36251e703557/volumes" Sep 11 19:01:33.172947 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:33.172934 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f22b59d-2dac-4285-a5c0-f70d5637a826" path="/var/lib/kubelet/pods/8f22b59d-2dac-4285-a5c0-f70d5637a826/volumes" Sep 11 19:01:33.173259 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:33.173248 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e60f0eb1-1fde-4b39-980b-d347bef49899" path="/var/lib/kubelet/pods/e60f0eb1-1fde-4b39-980b-d347bef49899/volumes" Sep 11 19:01:33.490511 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:33.490433 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:33.794452 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:33.794421 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:33.897123 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:33.897085 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:34.675039 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:34.675012 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:35.453067 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:35.453028 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:35.990091 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:35.990055 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:36.089828 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:36.089784 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:36.190481 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:36.190446 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:37.078321 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.078288 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:37.834981 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.834950 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv"] Sep 11 19:01:37.835386 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.835372 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e60f0eb1-1fde-4b39-980b-d347bef49899" containerName="payload-pre-processing" Sep 11 19:01:37.835386 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.835385 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e60f0eb1-1fde-4b39-980b-d347bef49899" containerName="payload-pre-processing" Sep 11 19:01:37.835529 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.835393 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f22b59d-2dac-4285-a5c0-f70d5637a826" containerName="revoke-keys" Sep 11 19:01:37.835529 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.835399 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f22b59d-2dac-4285-a5c0-f70d5637a826" containerName="revoke-keys" Sep 11 19:01:37.835529 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.835413 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44930a7b-15c0-4f25-be75-36251e703557" containerName="payload-processing" Sep 11 19:01:37.835529 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.835418 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="44930a7b-15c0-4f25-be75-36251e703557" containerName="payload-processing" Sep 11 19:01:37.835529 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.835479 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e60f0eb1-1fde-4b39-980b-d347bef49899" containerName="payload-pre-processing" Sep 11 19:01:37.835529 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.835487 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f22b59d-2dac-4285-a5c0-f70d5637a826" containerName="revoke-keys" Sep 11 19:01:37.835529 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.835493 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="44930a7b-15c0-4f25-be75-36251e703557" containerName="payload-processing" Sep 11 19:01:37.840095 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.840075 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:01:37.842944 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.842921 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 19:01:37.843051 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.842955 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6bftc\"" Sep 11 19:01:37.843051 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.842930 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-c158570a-serving-cert\"" Sep 11 19:01:37.847335 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.847314 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv"] Sep 11 19:01:37.873411 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.873389 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:37.889259 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.889228 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4p2j\" (UniqueName: \"kubernetes.io/projected/b9d2eda1-cedc-426e-bbd9-94ca016fc921-kube-api-access-w4p2j\") pod \"maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv\" (UID: \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:01:37.889366 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.889282 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9d2eda1-cedc-426e-bbd9-94ca016fc921-metrics-certs\") pod \"maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv\" (UID: \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:01:37.889419 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.889375 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d2eda1-cedc-426e-bbd9-94ca016fc921-openshift-service-ca\") pod \"maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv\" (UID: \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:01:37.889419 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.889408 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9d2eda1-cedc-426e-bbd9-94ca016fc921-maas-api-tls\") pod \"maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv\" (UID: \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:01:37.975273 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.975234 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:37.990442 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.990411 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d2eda1-cedc-426e-bbd9-94ca016fc921-openshift-service-ca\") pod \"maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv\" (UID: \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:01:37.990442 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.990441 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9d2eda1-cedc-426e-bbd9-94ca016fc921-maas-api-tls\") pod \"maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv\" (UID: \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:01:37.990671 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.990500 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4p2j\" (UniqueName: \"kubernetes.io/projected/b9d2eda1-cedc-426e-bbd9-94ca016fc921-kube-api-access-w4p2j\") pod \"maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv\" (UID: \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:01:37.990671 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.990563 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9d2eda1-cedc-426e-bbd9-94ca016fc921-metrics-certs\") pod \"maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv\" (UID: \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:01:37.991124 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.991092 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d2eda1-cedc-426e-bbd9-94ca016fc921-openshift-service-ca\") pod \"maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv\" (UID: \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:01:37.993133 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.993103 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9d2eda1-cedc-426e-bbd9-94ca016fc921-metrics-certs\") pod \"maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv\" (UID: \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:01:37.993238 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.993135 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9d2eda1-cedc-426e-bbd9-94ca016fc921-maas-api-tls\") pod \"maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv\" (UID: \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:01:37.998630 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:37.998604 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4p2j\" (UniqueName: \"kubernetes.io/projected/b9d2eda1-cedc-426e-bbd9-94ca016fc921-kube-api-access-w4p2j\") pod \"maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv\" (UID: \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:01:38.073910 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.073878 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:38.159511 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.159436 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:01:38.376973 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.376939 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:38.438717 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.438626 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv"] Sep 11 19:01:38.443365 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:01:38.443288 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d2eda1_cedc_426e_bbd9_94ca016fc921.slice/crio-fe5899bb2467345a0139aaf4c9a72f6a59215f3559cf9b56748d866a95086fd9 WatchSource:0}: Error finding container fe5899bb2467345a0139aaf4c9a72f6a59215f3559cf9b56748d866a95086fd9: Status 404 returned error can't find the container with id fe5899bb2467345a0139aaf4c9a72f6a59215f3559cf9b56748d866a95086fd9 Sep 11 19:01:38.475443 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.475409 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:38.525136 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.525105 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm"] Sep 11 19:01:38.529854 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.529837 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" Sep 11 19:01:38.532193 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.532167 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c158570a-dockercfg-snq79\"" Sep 11 19:01:38.532386 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.532372 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c158570a\"" Sep 11 19:01:38.536207 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.536187 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm"] Sep 11 19:01:38.594893 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.594856 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s588l\" (UniqueName: \"kubernetes.io/projected/89cec715-fbf0-46d4-8a40-1a8bc1cc63ee-kube-api-access-s588l\") pod \"payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm\" (UID: \"89cec715-fbf0-46d4-8a40-1a8bc1cc63ee\") " pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" Sep 11 19:01:38.595058 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.594902 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cec715-fbf0-46d4-8a40-1a8bc1cc63ee-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm\" (UID: \"89cec715-fbf0-46d4-8a40-1a8bc1cc63ee\") " pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" Sep 11 19:01:38.696024 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.695935 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s588l\" (UniqueName: \"kubernetes.io/projected/89cec715-fbf0-46d4-8a40-1a8bc1cc63ee-kube-api-access-s588l\") pod \"payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm\" (UID: \"89cec715-fbf0-46d4-8a40-1a8bc1cc63ee\") " pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" Sep 11 19:01:38.696024 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.695981 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cec715-fbf0-46d4-8a40-1a8bc1cc63ee-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm\" (UID: \"89cec715-fbf0-46d4-8a40-1a8bc1cc63ee\") " pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" Sep 11 19:01:38.696656 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.696638 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cec715-fbf0-46d4-8a40-1a8bc1cc63ee-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm\" (UID: \"89cec715-fbf0-46d4-8a40-1a8bc1cc63ee\") " pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" Sep 11 19:01:38.704695 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.704670 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s588l\" (UniqueName: \"kubernetes.io/projected/89cec715-fbf0-46d4-8a40-1a8bc1cc63ee-kube-api-access-s588l\") pod \"payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm\" (UID: \"89cec715-fbf0-46d4-8a40-1a8bc1cc63ee\") " pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" Sep 11 19:01:38.841797 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.841749 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" Sep 11 19:01:38.973441 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:38.973368 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" event={"ID":"b9d2eda1-cedc-426e-bbd9-94ca016fc921","Type":"ContainerStarted","Data":"fe5899bb2467345a0139aaf4c9a72f6a59215f3559cf9b56748d866a95086fd9"} Sep 11 19:01:39.003025 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:39.002987 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:39.127360 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:39.127320 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm"] Sep 11 19:01:39.128632 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:01:39.128598 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cec715_fbf0_46d4_8a40_1a8bc1cc63ee.slice/crio-d6953f00ff2097bab315b6c6d862209d51dede57ea73dc2e54d41fb9ecf3e752 WatchSource:0}: Error finding container d6953f00ff2097bab315b6c6d862209d51dede57ea73dc2e54d41fb9ecf3e752: Status 404 returned error can't find the container with id d6953f00ff2097bab315b6c6d862209d51dede57ea73dc2e54d41fb9ecf3e752 Sep 11 19:01:39.892470 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:39.892431 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:39.982386 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:39.982348 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" event={"ID":"89cec715-fbf0-46d4-8a40-1a8bc1cc63ee","Type":"ContainerStarted","Data":"3fb7087daf4c33a05615312bdca5bfba1dcef6376edfa17eefcc981326befa95"} Sep 11 19:01:39.982585 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:39.982407 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" Sep 11 19:01:39.982585 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:39.982424 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" event={"ID":"89cec715-fbf0-46d4-8a40-1a8bc1cc63ee","Type":"ContainerStarted","Data":"d6953f00ff2097bab315b6c6d862209d51dede57ea73dc2e54d41fb9ecf3e752"} Sep 11 19:01:39.999209 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:39.999141 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" podStartSLOduration=1.999121215 podStartE2EDuration="1.999121215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:01:39.998284287 +0000 UTC m=+805.452090261" watchObservedRunningTime="2026-09-11 19:01:39.999121215 +0000 UTC m=+805.452927171" Sep 11 19:01:40.615839 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:40.615792 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:40.928407 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:40.928140 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:40.990253 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:40.990218 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" event={"ID":"b9d2eda1-cedc-426e-bbd9-94ca016fc921","Type":"ContainerStarted","Data":"704be7ceb93695d3fad62f63156c75299c0f203387d66bbcca685c917c77cb89"} Sep 11 19:01:40.990635 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:40.990609 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:01:41.013989 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:41.013928 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" podStartSLOduration=1.5854070729999998 podStartE2EDuration="4.013907702s" totalImagesPullingTime="2.428500629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:01:41.011745804 +0000 UTC m=+806.465551787" watchObservedRunningTime="2026-09-11 19:01:41.013907702 +0000 UTC m=+806.467713657" Sep 11 19:01:41.480448 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:41.480407 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:01:47.001479 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:47.001447 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:01:50.993133 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:01:50.993104 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" Sep 11 19:02:00.236585 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:00.236526 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:01.235297 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:01.235251 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:02.079973 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:02.079932 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:02.856080 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:02.856045 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:03.751790 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:03.751760 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:04.452853 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:04.452817 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:05.161732 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:05.161694 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:05.982494 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:05.982457 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:06.841560 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:06.841509 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:17.173425 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:17.173393 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:17.744717 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:17.744678 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv"] Sep 11 19:02:17.744972 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:17.744945 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" podUID="b9d2eda1-cedc-426e-bbd9-94ca016fc921" containerName="maas-api" containerID="cri-o://704be7ceb93695d3fad62f63156c75299c0f203387d66bbcca685c917c77cb89" gracePeriod=30 Sep 11 19:02:17.776806 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:17.776777 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:17.802962 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:17.802929 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm"] Sep 11 19:02:17.803198 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:17.803173 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" podUID="89cec715-fbf0-46d4-8a40-1a8bc1cc63ee" containerName="payload-processing" containerID="cri-o://3fb7087daf4c33a05615312bdca5bfba1dcef6376edfa17eefcc981326befa95" gracePeriod=30 Sep 11 19:02:17.876725 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:17.876702 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:17.978275 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:17.978205 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:18.062813 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.062783 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:02:18.079907 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.079478 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:18.096032 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.095989 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" Sep 11 19:02:18.147501 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.147462 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9d2eda1-cedc-426e-bbd9-94ca016fc921-maas-api-tls\") pod \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\" (UID: \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\") " Sep 11 19:02:18.147687 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.147611 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4p2j\" (UniqueName: \"kubernetes.io/projected/b9d2eda1-cedc-426e-bbd9-94ca016fc921-kube-api-access-w4p2j\") pod \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\" (UID: \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\") " Sep 11 19:02:18.147687 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.147680 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9d2eda1-cedc-426e-bbd9-94ca016fc921-metrics-certs\") pod \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\" (UID: \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\") " Sep 11 19:02:18.147818 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.147774 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d2eda1-cedc-426e-bbd9-94ca016fc921-openshift-service-ca\") pod \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\" (UID: \"b9d2eda1-cedc-426e-bbd9-94ca016fc921\") " Sep 11 19:02:18.148483 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.148452 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d2eda1-cedc-426e-bbd9-94ca016fc921-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b9d2eda1-cedc-426e-bbd9-94ca016fc921" (UID: "b9d2eda1-cedc-426e-bbd9-94ca016fc921"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:02:18.149905 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.149819 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:18.150370 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.150344 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d2eda1-cedc-426e-bbd9-94ca016fc921-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b9d2eda1-cedc-426e-bbd9-94ca016fc921" (UID: "b9d2eda1-cedc-426e-bbd9-94ca016fc921"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:02:18.150370 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.150357 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d2eda1-cedc-426e-bbd9-94ca016fc921-kube-api-access-w4p2j" (OuterVolumeSpecName: "kube-api-access-w4p2j") pod "b9d2eda1-cedc-426e-bbd9-94ca016fc921" (UID: "b9d2eda1-cedc-426e-bbd9-94ca016fc921"). InnerVolumeSpecName "kube-api-access-w4p2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:02:18.150754 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.150735 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d2eda1-cedc-426e-bbd9-94ca016fc921-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b9d2eda1-cedc-426e-bbd9-94ca016fc921" (UID: "b9d2eda1-cedc-426e-bbd9-94ca016fc921"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:02:18.187444 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.187416 2556 generic.go:356] "Generic (PLEG): container finished" podID="89cec715-fbf0-46d4-8a40-1a8bc1cc63ee" containerID="3fb7087daf4c33a05615312bdca5bfba1dcef6376edfa17eefcc981326befa95" exitCode=0 Sep 11 19:02:18.187841 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.187454 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" Sep 11 19:02:18.187841 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.187500 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" event={"ID":"89cec715-fbf0-46d4-8a40-1a8bc1cc63ee","Type":"ContainerDied","Data":"3fb7087daf4c33a05615312bdca5bfba1dcef6376edfa17eefcc981326befa95"} Sep 11 19:02:18.187841 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.187590 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm" event={"ID":"89cec715-fbf0-46d4-8a40-1a8bc1cc63ee","Type":"ContainerDied","Data":"d6953f00ff2097bab315b6c6d862209d51dede57ea73dc2e54d41fb9ecf3e752"} Sep 11 19:02:18.187841 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.187628 2556 scope.go:117] "RemoveContainer" containerID="3fb7087daf4c33a05615312bdca5bfba1dcef6376edfa17eefcc981326befa95" Sep 11 19:02:18.188928 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.188908 2556 generic.go:356] "Generic (PLEG): container finished" podID="b9d2eda1-cedc-426e-bbd9-94ca016fc921" containerID="704be7ceb93695d3fad62f63156c75299c0f203387d66bbcca685c917c77cb89" exitCode=0 Sep 11 19:02:18.189036 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.188956 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" Sep 11 19:02:18.189036 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.188994 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" event={"ID":"b9d2eda1-cedc-426e-bbd9-94ca016fc921","Type":"ContainerDied","Data":"704be7ceb93695d3fad62f63156c75299c0f203387d66bbcca685c917c77cb89"} Sep 11 19:02:18.189140 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.189042 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv" event={"ID":"b9d2eda1-cedc-426e-bbd9-94ca016fc921","Type":"ContainerDied","Data":"fe5899bb2467345a0139aaf4c9a72f6a59215f3559cf9b56748d866a95086fd9"} Sep 11 19:02:18.198313 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.198299 2556 scope.go:117] "RemoveContainer" containerID="3fb7087daf4c33a05615312bdca5bfba1dcef6376edfa17eefcc981326befa95" Sep 11 19:02:18.198662 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:02:18.198638 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb7087daf4c33a05615312bdca5bfba1dcef6376edfa17eefcc981326befa95\": container with ID starting with 3fb7087daf4c33a05615312bdca5bfba1dcef6376edfa17eefcc981326befa95 not found: ID does not exist" containerID="3fb7087daf4c33a05615312bdca5bfba1dcef6376edfa17eefcc981326befa95" Sep 11 19:02:18.198746 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.198675 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb7087daf4c33a05615312bdca5bfba1dcef6376edfa17eefcc981326befa95"} err="failed to get container status \"3fb7087daf4c33a05615312bdca5bfba1dcef6376edfa17eefcc981326befa95\": rpc error: code = NotFound desc = could not find container \"3fb7087daf4c33a05615312bdca5bfba1dcef6376edfa17eefcc981326befa95\": container with ID starting with 3fb7087daf4c33a05615312bdca5bfba1dcef6376edfa17eefcc981326befa95 not found: ID does not exist" Sep 11 19:02:18.198746 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.198731 2556 scope.go:117] "RemoveContainer" containerID="704be7ceb93695d3fad62f63156c75299c0f203387d66bbcca685c917c77cb89" Sep 11 19:02:18.207864 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.207841 2556 scope.go:117] "RemoveContainer" containerID="704be7ceb93695d3fad62f63156c75299c0f203387d66bbcca685c917c77cb89" Sep 11 19:02:18.208232 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:02:18.208198 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704be7ceb93695d3fad62f63156c75299c0f203387d66bbcca685c917c77cb89\": container with ID starting with 704be7ceb93695d3fad62f63156c75299c0f203387d66bbcca685c917c77cb89 not found: ID does not exist" containerID="704be7ceb93695d3fad62f63156c75299c0f203387d66bbcca685c917c77cb89" Sep 11 19:02:18.208347 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.208243 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704be7ceb93695d3fad62f63156c75299c0f203387d66bbcca685c917c77cb89"} err="failed to get container status \"704be7ceb93695d3fad62f63156c75299c0f203387d66bbcca685c917c77cb89\": rpc error: code = NotFound desc = could not find container \"704be7ceb93695d3fad62f63156c75299c0f203387d66bbcca685c917c77cb89\": container with ID starting with 704be7ceb93695d3fad62f63156c75299c0f203387d66bbcca685c917c77cb89 not found: ID does not exist" Sep 11 19:02:18.210697 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.210661 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv"] Sep 11 19:02:18.218820 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.218798 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c158570a-6bb8b8b76-hrxbv"] Sep 11 19:02:18.248789 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.248765 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s588l\" (UniqueName: \"kubernetes.io/projected/89cec715-fbf0-46d4-8a40-1a8bc1cc63ee-kube-api-access-s588l\") pod \"89cec715-fbf0-46d4-8a40-1a8bc1cc63ee\" (UID: \"89cec715-fbf0-46d4-8a40-1a8bc1cc63ee\") " Sep 11 19:02:18.248890 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.248829 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cec715-fbf0-46d4-8a40-1a8bc1cc63ee-plugins-config-volume\") pod \"89cec715-fbf0-46d4-8a40-1a8bc1cc63ee\" (UID: \"89cec715-fbf0-46d4-8a40-1a8bc1cc63ee\") " Sep 11 19:02:18.249070 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.249050 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4p2j\" (UniqueName: \"kubernetes.io/projected/b9d2eda1-cedc-426e-bbd9-94ca016fc921-kube-api-access-w4p2j\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:02:18.249070 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.249070 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9d2eda1-cedc-426e-bbd9-94ca016fc921-metrics-certs\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:02:18.249231 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.249086 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d2eda1-cedc-426e-bbd9-94ca016fc921-openshift-service-ca\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:02:18.249231 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.249100 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9d2eda1-cedc-426e-bbd9-94ca016fc921-maas-api-tls\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:02:18.249298 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.249221 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89cec715-fbf0-46d4-8a40-1a8bc1cc63ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89cec715-fbf0-46d4-8a40-1a8bc1cc63ee" (UID: "89cec715-fbf0-46d4-8a40-1a8bc1cc63ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:02:18.250906 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.250884 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cec715-fbf0-46d4-8a40-1a8bc1cc63ee-kube-api-access-s588l" (OuterVolumeSpecName: "kube-api-access-s588l") pod "89cec715-fbf0-46d4-8a40-1a8bc1cc63ee" (UID: "89cec715-fbf0-46d4-8a40-1a8bc1cc63ee"). InnerVolumeSpecName "kube-api-access-s588l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:02:18.350313 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.350230 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cec715-fbf0-46d4-8a40-1a8bc1cc63ee-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:02:18.350313 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.350261 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s588l\" (UniqueName: \"kubernetes.io/projected/89cec715-fbf0-46d4-8a40-1a8bc1cc63ee-kube-api-access-s588l\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:02:18.458301 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.458258 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:18.512655 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.512620 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm"] Sep 11 19:02:18.518708 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.518677 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c158570a-7d585d68b-s9hpm"] Sep 11 19:02:18.570454 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.570422 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:18.879495 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:18.879423 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:19.066086 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:19.066050 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:19.172433 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:19.172399 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89cec715-fbf0-46d4-8a40-1a8bc1cc63ee" path="/var/lib/kubelet/pods/89cec715-fbf0-46d4-8a40-1a8bc1cc63ee/volumes" Sep 11 19:02:19.172908 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:19.172887 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9d2eda1-cedc-426e-bbd9-94ca016fc921" path="/var/lib/kubelet/pods/b9d2eda1-cedc-426e-bbd9-94ca016fc921/volumes" Sep 11 19:02:19.463250 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:19.463161 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:20.270467 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:20.270430 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:21.081459 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:21.081420 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:21.268925 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:21.268894 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:21.367116 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:21.367033 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:21.744117 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:21.744036 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:22.071200 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:22.071164 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:22.864470 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:22.864430 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:23.568115 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.568079 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f"] Sep 11 19:02:23.568787 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.568762 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89cec715-fbf0-46d4-8a40-1a8bc1cc63ee" containerName="payload-processing" Sep 11 19:02:23.568787 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.568785 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="89cec715-fbf0-46d4-8a40-1a8bc1cc63ee" containerName="payload-processing" Sep 11 19:02:23.568999 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.568810 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d2eda1-cedc-426e-bbd9-94ca016fc921" containerName="maas-api" Sep 11 19:02:23.568999 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.568820 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d2eda1-cedc-426e-bbd9-94ca016fc921" containerName="maas-api" Sep 11 19:02:23.568999 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.568917 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="89cec715-fbf0-46d4-8a40-1a8bc1cc63ee" containerName="payload-processing" Sep 11 19:02:23.568999 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.568932 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d2eda1-cedc-426e-bbd9-94ca016fc921" containerName="maas-api" Sep 11 19:02:23.577194 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.577172 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:23.581723 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.580209 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f0250e49-serving-cert\"" Sep 11 19:02:23.581723 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.580460 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6bftc\"" Sep 11 19:02:23.581723 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.581487 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 19:02:23.585081 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.584389 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f"] Sep 11 19:02:23.610364 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.610334 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:23.702913 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.702880 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44644b6-0d70-42c5-8edd-d925465df3f4-openshift-service-ca\") pod \"maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:23.703059 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.702948 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44644b6-0d70-42c5-8edd-d925465df3f4-metrics-certs\") pod \"maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:23.703059 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.702998 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c44644b6-0d70-42c5-8edd-d925465df3f4-maas-api-tls\") pod \"maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:23.703137 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.703092 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbgrj\" (UniqueName: \"kubernetes.io/projected/c44644b6-0d70-42c5-8edd-d925465df3f4-kube-api-access-mbgrj\") pod \"maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:23.710299 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.710272 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:23.804470 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.804432 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44644b6-0d70-42c5-8edd-d925465df3f4-metrics-certs\") pod \"maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:23.804470 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.804470 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c44644b6-0d70-42c5-8edd-d925465df3f4-maas-api-tls\") pod \"maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:23.804713 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.804515 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbgrj\" (UniqueName: \"kubernetes.io/projected/c44644b6-0d70-42c5-8edd-d925465df3f4-kube-api-access-mbgrj\") pod \"maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:23.804713 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.804606 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44644b6-0d70-42c5-8edd-d925465df3f4-openshift-service-ca\") pod \"maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:23.804713 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:02:23.804638 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-serving-cert: secret "maas-api-e2e-ait-f0250e49-serving-cert" not found Sep 11 19:02:23.804713 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:02:23.804706 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c44644b6-0d70-42c5-8edd-d925465df3f4-maas-api-tls podName:c44644b6-0d70-42c5-8edd-d925465df3f4 nodeName:}" failed. No retries permitted until 2026-09-11 19:02:24.304685566 +0000 UTC m=+849.758491516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c44644b6-0d70-42c5-8edd-d925465df3f4-maas-api-tls") pod "maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" (UID: "c44644b6-0d70-42c5-8edd-d925465df3f4") : secret "maas-api-e2e-ait-f0250e49-serving-cert" not found Sep 11 19:02:23.805245 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.805222 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44644b6-0d70-42c5-8edd-d925465df3f4-openshift-service-ca\") pod \"maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:23.806981 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.806959 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44644b6-0d70-42c5-8edd-d925465df3f4-metrics-certs\") pod \"maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:23.817913 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.817885 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbgrj\" (UniqueName: \"kubernetes.io/projected/c44644b6-0d70-42c5-8edd-d925465df3f4-kube-api-access-mbgrj\") pod \"maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:23.929588 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:23.929467 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:24.093746 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.093714 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:24.192059 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.191987 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:24.208218 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.208190 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x"] Sep 11 19:02:24.211914 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.211898 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" Sep 11 19:02:24.214295 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.214276 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f0250e49-dockercfg-4qfhf\"" Sep 11 19:02:24.214700 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.214678 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f0250e49\"" Sep 11 19:02:24.224839 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.224821 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x"] Sep 11 19:02:24.227853 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.227830 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94"] Sep 11 19:02:24.240750 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.240728 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" Sep 11 19:02:24.245422 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.245399 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94"] Sep 11 19:02:24.310328 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.310274 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c44644b6-0d70-42c5-8edd-d925465df3f4-maas-api-tls\") pod \"maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:24.310499 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.310343 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de55d743-86aa-43ba-90c2-8ab14c322112-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x\" (UID: \"de55d743-86aa-43ba-90c2-8ab14c322112\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" Sep 11 19:02:24.310499 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.310384 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd6jg\" (UniqueName: \"kubernetes.io/projected/de55d743-86aa-43ba-90c2-8ab14c322112-kube-api-access-qd6jg\") pod \"payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x\" (UID: \"de55d743-86aa-43ba-90c2-8ab14c322112\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" Sep 11 19:02:24.312954 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.312925 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c44644b6-0d70-42c5-8edd-d925465df3f4-maas-api-tls\") pod \"maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:24.411233 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.411200 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc06803-b683-49b8-beda-105b302644cf-plugins-config-volume\") pod \"payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94\" (UID: \"1cc06803-b683-49b8-beda-105b302644cf\") " pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" Sep 11 19:02:24.411419 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.411298 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de55d743-86aa-43ba-90c2-8ab14c322112-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x\" (UID: \"de55d743-86aa-43ba-90c2-8ab14c322112\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" Sep 11 19:02:24.411419 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.411345 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd6jg\" (UniqueName: \"kubernetes.io/projected/de55d743-86aa-43ba-90c2-8ab14c322112-kube-api-access-qd6jg\") pod \"payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x\" (UID: \"de55d743-86aa-43ba-90c2-8ab14c322112\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" Sep 11 19:02:24.411419 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.411386 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m9s6\" (UniqueName: \"kubernetes.io/projected/1cc06803-b683-49b8-beda-105b302644cf-kube-api-access-9m9s6\") pod \"payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94\" (UID: \"1cc06803-b683-49b8-beda-105b302644cf\") " pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" Sep 11 19:02:24.412024 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.412005 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de55d743-86aa-43ba-90c2-8ab14c322112-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x\" (UID: \"de55d743-86aa-43ba-90c2-8ab14c322112\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" Sep 11 19:02:24.419159 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.419141 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd6jg\" (UniqueName: \"kubernetes.io/projected/de55d743-86aa-43ba-90c2-8ab14c322112-kube-api-access-qd6jg\") pod \"payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x\" (UID: \"de55d743-86aa-43ba-90c2-8ab14c322112\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" Sep 11 19:02:24.494001 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.493927 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:24.512412 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.512375 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc06803-b683-49b8-beda-105b302644cf-plugins-config-volume\") pod \"payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94\" (UID: \"1cc06803-b683-49b8-beda-105b302644cf\") " pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" Sep 11 19:02:24.512559 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.512459 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m9s6\" (UniqueName: \"kubernetes.io/projected/1cc06803-b683-49b8-beda-105b302644cf-kube-api-access-9m9s6\") pod \"payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94\" (UID: \"1cc06803-b683-49b8-beda-105b302644cf\") " pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" Sep 11 19:02:24.513380 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.513350 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc06803-b683-49b8-beda-105b302644cf-plugins-config-volume\") pod \"payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94\" (UID: \"1cc06803-b683-49b8-beda-105b302644cf\") " pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" Sep 11 19:02:24.520231 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.520205 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m9s6\" (UniqueName: \"kubernetes.io/projected/1cc06803-b683-49b8-beda-105b302644cf-kube-api-access-9m9s6\") pod \"payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94\" (UID: \"1cc06803-b683-49b8-beda-105b302644cf\") " pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" Sep 11 19:02:24.523049 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.523025 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" Sep 11 19:02:24.554949 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.554745 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" Sep 11 19:02:24.675068 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.675037 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:24.772891 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.772870 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f"] Sep 11 19:02:24.775234 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:02:24.775204 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44644b6_0d70_42c5_8edd_d925465df3f4.slice/crio-f9d41dca53ea8d8e327598889657942b2425b6ed855daff951e74133d81585ad WatchSource:0}: Error finding container f9d41dca53ea8d8e327598889657942b2425b6ed855daff951e74133d81585ad: Status 404 returned error can't find the container with id f9d41dca53ea8d8e327598889657942b2425b6ed855daff951e74133d81585ad Sep 11 19:02:24.818889 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.818858 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x"] Sep 11 19:02:24.820405 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:02:24.820378 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde55d743_86aa_43ba_90c2_8ab14c322112.slice/crio-7adefdee127187b514491b7ed880eca5062960fdd910a3a7ce7d7644f99c1db6 WatchSource:0}: Error finding container 7adefdee127187b514491b7ed880eca5062960fdd910a3a7ce7d7644f99c1db6: Status 404 returned error can't find the container with id 7adefdee127187b514491b7ed880eca5062960fdd910a3a7ce7d7644f99c1db6 Sep 11 19:02:24.848567 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:24.848531 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94"] Sep 11 19:02:24.858678 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:02:24.858650 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc06803_b683_49b8_beda_105b302644cf.slice/crio-31168045e5b4c1ec1daabca0996d44afc615936fdc73964b439c167440c97668 WatchSource:0}: Error finding container 31168045e5b4c1ec1daabca0996d44afc615936fdc73964b439c167440c97668: Status 404 returned error can't find the container with id 31168045e5b4c1ec1daabca0996d44afc615936fdc73964b439c167440c97668 Sep 11 19:02:25.234570 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:25.234414 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" event={"ID":"1cc06803-b683-49b8-beda-105b302644cf","Type":"ContainerStarted","Data":"388d328c764e04af661f13d3b3b86de977eac8438e85beaa2cd49b96c64601e2"} Sep 11 19:02:25.234570 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:25.234484 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" Sep 11 19:02:25.234570 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:25.234513 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" event={"ID":"1cc06803-b683-49b8-beda-105b302644cf","Type":"ContainerStarted","Data":"31168045e5b4c1ec1daabca0996d44afc615936fdc73964b439c167440c97668"} Sep 11 19:02:25.236312 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:25.236285 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" event={"ID":"de55d743-86aa-43ba-90c2-8ab14c322112","Type":"ContainerStarted","Data":"110ac854faa8367aee226ebd5ecc03e62be75ce176c1560f6d2f8a39e4857ab9"} Sep 11 19:02:25.236476 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:25.236457 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" event={"ID":"de55d743-86aa-43ba-90c2-8ab14c322112","Type":"ContainerStarted","Data":"7adefdee127187b514491b7ed880eca5062960fdd910a3a7ce7d7644f99c1db6"} Sep 11 19:02:25.236632 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:25.236616 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" Sep 11 19:02:25.238146 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:25.238105 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" event={"ID":"c44644b6-0d70-42c5-8edd-d925465df3f4","Type":"ContainerStarted","Data":"8e00213ff7925e7fb92ad980d6fdd59c0d8481d226e8cac857df3b6289a8465e"} Sep 11 19:02:25.238244 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:25.238170 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:25.238244 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:25.238194 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" event={"ID":"c44644b6-0d70-42c5-8edd-d925465df3f4","Type":"ContainerStarted","Data":"f9d41dca53ea8d8e327598889657942b2425b6ed855daff951e74133d81585ad"} Sep 11 19:02:25.254440 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:25.251845 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" podStartSLOduration=1.251828787 podStartE2EDuration="1.251828787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:02:25.249373593 +0000 UTC m=+850.703179558" watchObservedRunningTime="2026-09-11 19:02:25.251828787 +0000 UTC m=+850.705634742" Sep 11 19:02:25.265187 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:25.265138 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" podStartSLOduration=1.2651203309999999 podStartE2EDuration="1.265120331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:02:25.263470923 +0000 UTC m=+850.717276896" watchObservedRunningTime="2026-09-11 19:02:25.265120331 +0000 UTC m=+850.718926287" Sep 11 19:02:25.279661 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:25.279621 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" podStartSLOduration=1.892288857 podStartE2EDuration="2.279608822s" totalImagesPullingTime="387.319965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:02:25.278531764 +0000 UTC m=+850.732337739" watchObservedRunningTime="2026-09-11 19:02:25.279608822 +0000 UTC m=+850.733414815" Sep 11 19:02:25.470091 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:25.470006 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:26.172363 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:26.172306 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:26.886279 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:26.886244 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:31.249896 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:31.249870 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:36.247238 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:36.247206 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" Sep 11 19:02:37.252075 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:37.252049 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" Sep 11 19:02:39.124289 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:39.124257 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:45.768404 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:45.768365 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:46.485042 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:46.485008 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:47.387174 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:47.387127 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:48.277783 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:48.277725 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:48.964746 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:48.964715 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:49.658710 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:49.658668 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:50.533022 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:50.532981 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:51.262690 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:51.262660 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:52.003850 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:52.003809 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:52.679393 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:52.679353 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f"] Sep 11 19:02:52.679658 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:52.679628 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" podUID="c44644b6-0d70-42c5-8edd-d925465df3f4" containerName="maas-api" containerID="cri-o://8e00213ff7925e7fb92ad980d6fdd59c0d8481d226e8cac857df3b6289a8465e" gracePeriod=30 Sep 11 19:02:52.709685 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:52.709656 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:52.742766 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:52.742730 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94"] Sep 11 19:02:52.743480 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:52.743451 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" podUID="1cc06803-b683-49b8-beda-105b302644cf" containerName="payload-processing" containerID="cri-o://388d328c764e04af661f13d3b3b86de977eac8438e85beaa2cd49b96c64601e2" gracePeriod=30 Sep 11 19:02:52.752740 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:52.752697 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x"] Sep 11 19:02:52.753934 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:52.752988 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" podUID="de55d743-86aa-43ba-90c2-8ab14c322112" containerName="payload-pre-processing" containerID="cri-o://110ac854faa8367aee226ebd5ecc03e62be75ce176c1560f6d2f8a39e4857ab9" gracePeriod=30 Sep 11 19:02:52.786743 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:52.786712 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:52.852824 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:52.852796 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:52.932794 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:52.932680 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:52.993118 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:52.993083 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:53.020382 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.020354 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:53.062563 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.062481 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44644b6-0d70-42c5-8edd-d925465df3f4-openshift-service-ca\") pod \"c44644b6-0d70-42c5-8edd-d925465df3f4\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " Sep 11 19:02:53.062700 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.062629 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbgrj\" (UniqueName: \"kubernetes.io/projected/c44644b6-0d70-42c5-8edd-d925465df3f4-kube-api-access-mbgrj\") pod \"c44644b6-0d70-42c5-8edd-d925465df3f4\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " Sep 11 19:02:53.062700 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.062662 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44644b6-0d70-42c5-8edd-d925465df3f4-metrics-certs\") pod \"c44644b6-0d70-42c5-8edd-d925465df3f4\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " Sep 11 19:02:53.062700 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.062697 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c44644b6-0d70-42c5-8edd-d925465df3f4-maas-api-tls\") pod \"c44644b6-0d70-42c5-8edd-d925465df3f4\" (UID: \"c44644b6-0d70-42c5-8edd-d925465df3f4\") " Sep 11 19:02:53.062922 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.062787 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44644b6-0d70-42c5-8edd-d925465df3f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c44644b6-0d70-42c5-8edd-d925465df3f4" (UID: "c44644b6-0d70-42c5-8edd-d925465df3f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:02:53.063104 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.063079 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44644b6-0d70-42c5-8edd-d925465df3f4-openshift-service-ca\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:02:53.063498 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.063478 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" Sep 11 19:02:53.065165 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.065131 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c44644b6-0d70-42c5-8edd-d925465df3f4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c44644b6-0d70-42c5-8edd-d925465df3f4" (UID: "c44644b6-0d70-42c5-8edd-d925465df3f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:02:53.065371 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.065338 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c44644b6-0d70-42c5-8edd-d925465df3f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c44644b6-0d70-42c5-8edd-d925465df3f4" (UID: "c44644b6-0d70-42c5-8edd-d925465df3f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:02:53.065514 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.065493 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44644b6-0d70-42c5-8edd-d925465df3f4-kube-api-access-mbgrj" (OuterVolumeSpecName: "kube-api-access-mbgrj") pod "c44644b6-0d70-42c5-8edd-d925465df3f4" (UID: "c44644b6-0d70-42c5-8edd-d925465df3f4"). InnerVolumeSpecName "kube-api-access-mbgrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:02:53.067942 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.067925 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" Sep 11 19:02:53.164174 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.164146 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qd6jg\" (UniqueName: \"kubernetes.io/projected/de55d743-86aa-43ba-90c2-8ab14c322112-kube-api-access-qd6jg\") pod \"de55d743-86aa-43ba-90c2-8ab14c322112\" (UID: \"de55d743-86aa-43ba-90c2-8ab14c322112\") " Sep 11 19:02:53.164355 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.164179 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc06803-b683-49b8-beda-105b302644cf-plugins-config-volume\") pod \"1cc06803-b683-49b8-beda-105b302644cf\" (UID: \"1cc06803-b683-49b8-beda-105b302644cf\") " Sep 11 19:02:53.164355 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.164243 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de55d743-86aa-43ba-90c2-8ab14c322112-plugins-config-volume\") pod \"de55d743-86aa-43ba-90c2-8ab14c322112\" (UID: \"de55d743-86aa-43ba-90c2-8ab14c322112\") " Sep 11 19:02:53.164355 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.164269 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m9s6\" (UniqueName: \"kubernetes.io/projected/1cc06803-b683-49b8-beda-105b302644cf-kube-api-access-9m9s6\") pod \"1cc06803-b683-49b8-beda-105b302644cf\" (UID: \"1cc06803-b683-49b8-beda-105b302644cf\") " Sep 11 19:02:53.164670 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.164639 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de55d743-86aa-43ba-90c2-8ab14c322112-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "de55d743-86aa-43ba-90c2-8ab14c322112" (UID: "de55d743-86aa-43ba-90c2-8ab14c322112"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:02:53.164670 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.164645 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbgrj\" (UniqueName: \"kubernetes.io/projected/c44644b6-0d70-42c5-8edd-d925465df3f4-kube-api-access-mbgrj\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:02:53.164884 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.164690 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c44644b6-0d70-42c5-8edd-d925465df3f4-metrics-certs\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:02:53.164884 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.164703 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c44644b6-0d70-42c5-8edd-d925465df3f4-maas-api-tls\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:02:53.164884 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.164652 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc06803-b683-49b8-beda-105b302644cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1cc06803-b683-49b8-beda-105b302644cf" (UID: "1cc06803-b683-49b8-beda-105b302644cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:02:53.166416 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.166397 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc06803-b683-49b8-beda-105b302644cf-kube-api-access-9m9s6" (OuterVolumeSpecName: "kube-api-access-9m9s6") pod "1cc06803-b683-49b8-beda-105b302644cf" (UID: "1cc06803-b683-49b8-beda-105b302644cf"). InnerVolumeSpecName "kube-api-access-9m9s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:02:53.166690 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.166675 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de55d743-86aa-43ba-90c2-8ab14c322112-kube-api-access-qd6jg" (OuterVolumeSpecName: "kube-api-access-qd6jg") pod "de55d743-86aa-43ba-90c2-8ab14c322112" (UID: "de55d743-86aa-43ba-90c2-8ab14c322112"). InnerVolumeSpecName "kube-api-access-qd6jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:02:53.265412 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.265384 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de55d743-86aa-43ba-90c2-8ab14c322112-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:02:53.265412 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.265409 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m9s6\" (UniqueName: \"kubernetes.io/projected/1cc06803-b683-49b8-beda-105b302644cf-kube-api-access-9m9s6\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:02:53.265412 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.265418 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd6jg\" (UniqueName: \"kubernetes.io/projected/de55d743-86aa-43ba-90c2-8ab14c322112-kube-api-access-qd6jg\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:02:53.265624 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.265426 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc06803-b683-49b8-beda-105b302644cf-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:02:53.294074 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.294048 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:53.387633 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.387600 2556 generic.go:356] "Generic (PLEG): container finished" podID="1cc06803-b683-49b8-beda-105b302644cf" containerID="388d328c764e04af661f13d3b3b86de977eac8438e85beaa2cd49b96c64601e2" exitCode=0 Sep 11 19:02:53.387792 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.387641 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" Sep 11 19:02:53.387792 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.387692 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" event={"ID":"1cc06803-b683-49b8-beda-105b302644cf","Type":"ContainerDied","Data":"388d328c764e04af661f13d3b3b86de977eac8438e85beaa2cd49b96c64601e2"} Sep 11 19:02:53.387792 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.387750 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94" event={"ID":"1cc06803-b683-49b8-beda-105b302644cf","Type":"ContainerDied","Data":"31168045e5b4c1ec1daabca0996d44afc615936fdc73964b439c167440c97668"} Sep 11 19:02:53.387792 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.387782 2556 scope.go:117] "RemoveContainer" containerID="388d328c764e04af661f13d3b3b86de977eac8438e85beaa2cd49b96c64601e2" Sep 11 19:02:53.388989 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.388967 2556 generic.go:356] "Generic (PLEG): container finished" podID="de55d743-86aa-43ba-90c2-8ab14c322112" containerID="110ac854faa8367aee226ebd5ecc03e62be75ce176c1560f6d2f8a39e4857ab9" exitCode=0 Sep 11 19:02:53.389100 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.389019 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" Sep 11 19:02:53.389100 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.389067 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" event={"ID":"de55d743-86aa-43ba-90c2-8ab14c322112","Type":"ContainerDied","Data":"110ac854faa8367aee226ebd5ecc03e62be75ce176c1560f6d2f8a39e4857ab9"} Sep 11 19:02:53.389205 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.389113 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x" event={"ID":"de55d743-86aa-43ba-90c2-8ab14c322112","Type":"ContainerDied","Data":"7adefdee127187b514491b7ed880eca5062960fdd910a3a7ce7d7644f99c1db6"} Sep 11 19:02:53.390638 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.390615 2556 generic.go:356] "Generic (PLEG): container finished" podID="c44644b6-0d70-42c5-8edd-d925465df3f4" containerID="8e00213ff7925e7fb92ad980d6fdd59c0d8481d226e8cac857df3b6289a8465e" exitCode=0 Sep 11 19:02:53.390736 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.390659 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" Sep 11 19:02:53.390736 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.390683 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" event={"ID":"c44644b6-0d70-42c5-8edd-d925465df3f4","Type":"ContainerDied","Data":"8e00213ff7925e7fb92ad980d6fdd59c0d8481d226e8cac857df3b6289a8465e"} Sep 11 19:02:53.390736 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.390727 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f" event={"ID":"c44644b6-0d70-42c5-8edd-d925465df3f4","Type":"ContainerDied","Data":"f9d41dca53ea8d8e327598889657942b2425b6ed855daff951e74133d81585ad"} Sep 11 19:02:53.397705 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.397692 2556 scope.go:117] "RemoveContainer" containerID="388d328c764e04af661f13d3b3b86de977eac8438e85beaa2cd49b96c64601e2" Sep 11 19:02:53.397991 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:02:53.397971 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388d328c764e04af661f13d3b3b86de977eac8438e85beaa2cd49b96c64601e2\": container with ID starting with 388d328c764e04af661f13d3b3b86de977eac8438e85beaa2cd49b96c64601e2 not found: ID does not exist" containerID="388d328c764e04af661f13d3b3b86de977eac8438e85beaa2cd49b96c64601e2" Sep 11 19:02:53.398074 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.398004 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388d328c764e04af661f13d3b3b86de977eac8438e85beaa2cd49b96c64601e2"} err="failed to get container status \"388d328c764e04af661f13d3b3b86de977eac8438e85beaa2cd49b96c64601e2\": rpc error: code = NotFound desc = could not find container \"388d328c764e04af661f13d3b3b86de977eac8438e85beaa2cd49b96c64601e2\": container with ID starting with 388d328c764e04af661f13d3b3b86de977eac8438e85beaa2cd49b96c64601e2 not found: ID does not exist" Sep 11 19:02:53.398074 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.398026 2556 scope.go:117] "RemoveContainer" containerID="110ac854faa8367aee226ebd5ecc03e62be75ce176c1560f6d2f8a39e4857ab9" Sep 11 19:02:53.405735 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.405712 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94"] Sep 11 19:02:53.406272 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.406256 2556 scope.go:117] "RemoveContainer" containerID="110ac854faa8367aee226ebd5ecc03e62be75ce176c1560f6d2f8a39e4857ab9" Sep 11 19:02:53.406512 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:02:53.406494 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110ac854faa8367aee226ebd5ecc03e62be75ce176c1560f6d2f8a39e4857ab9\": container with ID starting with 110ac854faa8367aee226ebd5ecc03e62be75ce176c1560f6d2f8a39e4857ab9 not found: ID does not exist" containerID="110ac854faa8367aee226ebd5ecc03e62be75ce176c1560f6d2f8a39e4857ab9" Sep 11 19:02:53.406586 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.406518 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110ac854faa8367aee226ebd5ecc03e62be75ce176c1560f6d2f8a39e4857ab9"} err="failed to get container status \"110ac854faa8367aee226ebd5ecc03e62be75ce176c1560f6d2f8a39e4857ab9\": rpc error: code = NotFound desc = could not find container \"110ac854faa8367aee226ebd5ecc03e62be75ce176c1560f6d2f8a39e4857ab9\": container with ID starting with 110ac854faa8367aee226ebd5ecc03e62be75ce176c1560f6d2f8a39e4857ab9 not found: ID does not exist" Sep 11 19:02:53.406586 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.406534 2556 scope.go:117] "RemoveContainer" containerID="8e00213ff7925e7fb92ad980d6fdd59c0d8481d226e8cac857df3b6289a8465e" Sep 11 19:02:53.410579 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.410559 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0250e49-7d8b66b5d5-v5b94"] Sep 11 19:02:53.414836 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.414819 2556 scope.go:117] "RemoveContainer" containerID="8e00213ff7925e7fb92ad980d6fdd59c0d8481d226e8cac857df3b6289a8465e" Sep 11 19:02:53.415086 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:02:53.415069 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e00213ff7925e7fb92ad980d6fdd59c0d8481d226e8cac857df3b6289a8465e\": container with ID starting with 8e00213ff7925e7fb92ad980d6fdd59c0d8481d226e8cac857df3b6289a8465e not found: ID does not exist" containerID="8e00213ff7925e7fb92ad980d6fdd59c0d8481d226e8cac857df3b6289a8465e" Sep 11 19:02:53.415146 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.415091 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e00213ff7925e7fb92ad980d6fdd59c0d8481d226e8cac857df3b6289a8465e"} err="failed to get container status \"8e00213ff7925e7fb92ad980d6fdd59c0d8481d226e8cac857df3b6289a8465e\": rpc error: code = NotFound desc = could not find container \"8e00213ff7925e7fb92ad980d6fdd59c0d8481d226e8cac857df3b6289a8465e\": container with ID starting with 8e00213ff7925e7fb92ad980d6fdd59c0d8481d226e8cac857df3b6289a8465e not found: ID does not exist" Sep 11 19:02:53.422680 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.422660 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x"] Sep 11 19:02:53.425938 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.425918 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0250e49-546cf645d-mh69x"] Sep 11 19:02:53.437429 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.437385 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f"] Sep 11 19:02:53.439922 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:53.439905 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0250e49-68857d74b8-p4k2f"] Sep 11 19:02:54.007897 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:54.007793 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:54.208208 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:54.208174 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:54.443614 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:54.443584 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:54.732562 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:54.732475 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:55.006441 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:55.006354 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:55.172264 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:55.172233 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cc06803-b683-49b8-beda-105b302644cf" path="/var/lib/kubelet/pods/1cc06803-b683-49b8-beda-105b302644cf/volumes" Sep 11 19:02:55.172657 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:55.172643 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c44644b6-0d70-42c5-8edd-d925465df3f4" path="/var/lib/kubelet/pods/c44644b6-0d70-42c5-8edd-d925465df3f4/volumes" Sep 11 19:02:55.172959 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:55.172948 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de55d743-86aa-43ba-90c2-8ab14c322112" path="/var/lib/kubelet/pods/de55d743-86aa-43ba-90c2-8ab14c322112/volumes" Sep 11 19:02:55.342316 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:55.342285 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:55.865552 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:55.865507 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:56.666999 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.666965 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:56.746764 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.746734 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-89e163-fzbkj"] Sep 11 19:02:56.747201 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.747188 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c44644b6-0d70-42c5-8edd-d925465df3f4" containerName="maas-api" Sep 11 19:02:56.747255 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.747203 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c44644b6-0d70-42c5-8edd-d925465df3f4" containerName="maas-api" Sep 11 19:02:56.747255 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.747212 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de55d743-86aa-43ba-90c2-8ab14c322112" containerName="payload-pre-processing" Sep 11 19:02:56.747255 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.747217 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="de55d743-86aa-43ba-90c2-8ab14c322112" containerName="payload-pre-processing" Sep 11 19:02:56.747255 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.747229 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc06803-b683-49b8-beda-105b302644cf" containerName="payload-processing" Sep 11 19:02:56.747255 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.747237 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc06803-b683-49b8-beda-105b302644cf" containerName="payload-processing" Sep 11 19:02:56.747405 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.747306 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="de55d743-86aa-43ba-90c2-8ab14c322112" containerName="payload-pre-processing" Sep 11 19:02:56.747405 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.747321 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc06803-b683-49b8-beda-105b302644cf" containerName="payload-processing" Sep 11 19:02:56.747405 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.747327 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c44644b6-0d70-42c5-8edd-d925465df3f4" containerName="maas-api" Sep 11 19:02:56.751920 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.751902 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-89e163-fzbkj" Sep 11 19:02:56.754069 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.754046 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\"" Sep 11 19:02:56.758085 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.758063 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-89e163-fzbkj"] Sep 11 19:02:56.796189 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.796159 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d5b5cd-db84-42a6-bb59-b52711d62ce5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-89e163-fzbkj\" (UID: \"00d5b5cd-db84-42a6-bb59-b52711d62ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-89e163-fzbkj" Sep 11 19:02:56.897216 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.897189 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d5b5cd-db84-42a6-bb59-b52711d62ce5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-89e163-fzbkj\" (UID: \"00d5b5cd-db84-42a6-bb59-b52711d62ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-89e163-fzbkj" Sep 11 19:02:56.897760 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:56.897744 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d5b5cd-db84-42a6-bb59-b52711d62ce5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-89e163-fzbkj\" (UID: \"00d5b5cd-db84-42a6-bb59-b52711d62ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-89e163-fzbkj" Sep 11 19:02:57.065655 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:57.065618 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-89e163-fzbkj" Sep 11 19:02:57.345104 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:57.345077 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-89e163-fzbkj"] Sep 11 19:02:57.347178 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:02:57.347145 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d5b5cd_db84_42a6_bb59_b52711d62ce5.slice/crio-84cec3eabb93f36cb7b68af8b51785df2c59ba19c7617ac92f1228bf1b055912 WatchSource:0}: Error finding container 84cec3eabb93f36cb7b68af8b51785df2c59ba19c7617ac92f1228bf1b055912: Status 404 returned error can't find the container with id 84cec3eabb93f36cb7b68af8b51785df2c59ba19c7617ac92f1228bf1b055912 Sep 11 19:02:57.417706 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:57.417677 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-89e163-fzbkj" event={"ID":"00d5b5cd-db84-42a6-bb59-b52711d62ce5","Type":"ContainerStarted","Data":"84cec3eabb93f36cb7b68af8b51785df2c59ba19c7617ac92f1228bf1b055912"} Sep 11 19:02:57.550990 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:57.550953 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:57.651910 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:57.651879 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:58.415826 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:58.415735 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:58.424272 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:58.424244 2556 generic.go:356] "Generic (PLEG): container finished" podID="00d5b5cd-db84-42a6-bb59-b52711d62ce5" containerID="de41741bb5304ddbf6cdc5e9887d10bb52d2f5975c167c5a537c18fad8c7366e" exitCode=0 Sep 11 19:02:58.424413 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:58.424328 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-89e163-fzbkj" event={"ID":"00d5b5cd-db84-42a6-bb59-b52711d62ce5","Type":"ContainerDied","Data":"de41741bb5304ddbf6cdc5e9887d10bb52d2f5975c167c5a537c18fad8c7366e"} Sep 11 19:02:59.527998 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:59.527946 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:02:59.564969 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:59.564941 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-89e163-fzbkj" Sep 11 19:02:59.623826 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:59.623798 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d5b5cd-db84-42a6-bb59-b52711d62ce5-maas-api-service-ca\") pod \"00d5b5cd-db84-42a6-bb59-b52711d62ce5\" (UID: \"00d5b5cd-db84-42a6-bb59-b52711d62ce5\") " Sep 11 19:02:59.624207 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:59.624185 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d5b5cd-db84-42a6-bb59-b52711d62ce5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00d5b5cd-db84-42a6-bb59-b52711d62ce5" (UID: "00d5b5cd-db84-42a6-bb59-b52711d62ce5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:02:59.725443 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:02:59.725358 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d5b5cd-db84-42a6-bb59-b52711d62ce5-maas-api-service-ca\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:03:00.268449 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:00.268417 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:00.446558 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:00.446517 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-89e163-fzbkj" Sep 11 19:03:00.446558 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:00.446518 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-89e163-fzbkj" event={"ID":"00d5b5cd-db84-42a6-bb59-b52711d62ce5","Type":"ContainerDied","Data":"84cec3eabb93f36cb7b68af8b51785df2c59ba19c7617ac92f1228bf1b055912"} Sep 11 19:03:00.446780 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:00.446631 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84cec3eabb93f36cb7b68af8b51785df2c59ba19c7617ac92f1228bf1b055912" Sep 11 19:03:00.991552 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:00.991515 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:01.973053 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:01.973021 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:02.734127 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.734096 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6"] Sep 11 19:03:02.734616 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.734597 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d5b5cd-db84-42a6-bb59-b52711d62ce5" containerName="revoke-keys" Sep 11 19:03:02.734616 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.734612 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d5b5cd-db84-42a6-bb59-b52711d62ce5" containerName="revoke-keys" Sep 11 19:03:02.734731 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.734694 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d5b5cd-db84-42a6-bb59-b52711d62ce5" containerName="revoke-keys" Sep 11 19:03:02.738001 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.737981 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:02.740461 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.740420 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8092ee69-serving-cert\"" Sep 11 19:03:02.740908 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.740887 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 19:03:02.741014 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.740894 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6bftc\"" Sep 11 19:03:02.748090 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.748065 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6"] Sep 11 19:03:02.778186 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.778090 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:02.856108 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.855763 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5120d06d-7ee2-409a-b44f-5fdad92b4334-maas-api-tls\") pod \"maas-api-e2e-derive-8092ee69-7958f477b-lcdf6\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:02.856108 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.855846 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5120d06d-7ee2-409a-b44f-5fdad92b4334-openshift-service-ca\") pod \"maas-api-e2e-derive-8092ee69-7958f477b-lcdf6\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:02.856108 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.855893 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5120d06d-7ee2-409a-b44f-5fdad92b4334-metrics-certs\") pod \"maas-api-e2e-derive-8092ee69-7958f477b-lcdf6\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:02.856108 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.855934 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbs9t\" (UniqueName: \"kubernetes.io/projected/5120d06d-7ee2-409a-b44f-5fdad92b4334-kube-api-access-fbs9t\") pod \"maas-api-e2e-derive-8092ee69-7958f477b-lcdf6\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:02.879932 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.879905 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:02.956973 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.956945 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5120d06d-7ee2-409a-b44f-5fdad92b4334-maas-api-tls\") pod \"maas-api-e2e-derive-8092ee69-7958f477b-lcdf6\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:02.957107 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.956994 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5120d06d-7ee2-409a-b44f-5fdad92b4334-openshift-service-ca\") pod \"maas-api-e2e-derive-8092ee69-7958f477b-lcdf6\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:02.957107 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.957020 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5120d06d-7ee2-409a-b44f-5fdad92b4334-metrics-certs\") pod \"maas-api-e2e-derive-8092ee69-7958f477b-lcdf6\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:02.957107 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:03:02.957097 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-serving-cert: secret "maas-api-e2e-derive-8092ee69-serving-cert" not found Sep 11 19:03:02.957272 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:03:02.957164 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5120d06d-7ee2-409a-b44f-5fdad92b4334-maas-api-tls podName:5120d06d-7ee2-409a-b44f-5fdad92b4334 nodeName:}" failed. No retries permitted until 2026-09-11 19:03:03.457144486 +0000 UTC m=+888.910950436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5120d06d-7ee2-409a-b44f-5fdad92b4334-maas-api-tls") pod "maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" (UID: "5120d06d-7ee2-409a-b44f-5fdad92b4334") : secret "maas-api-e2e-derive-8092ee69-serving-cert" not found Sep 11 19:03:02.957272 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.957191 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbs9t\" (UniqueName: \"kubernetes.io/projected/5120d06d-7ee2-409a-b44f-5fdad92b4334-kube-api-access-fbs9t\") pod \"maas-api-e2e-derive-8092ee69-7958f477b-lcdf6\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:02.957736 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.957718 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5120d06d-7ee2-409a-b44f-5fdad92b4334-openshift-service-ca\") pod \"maas-api-e2e-derive-8092ee69-7958f477b-lcdf6\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:02.959405 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.959387 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5120d06d-7ee2-409a-b44f-5fdad92b4334-metrics-certs\") pod \"maas-api-e2e-derive-8092ee69-7958f477b-lcdf6\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:02.969067 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:02.969041 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbs9t\" (UniqueName: \"kubernetes.io/projected/5120d06d-7ee2-409a-b44f-5fdad92b4334-kube-api-access-fbs9t\") pod \"maas-api-e2e-derive-8092ee69-7958f477b-lcdf6\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:03.100286 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.100253 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:03.287157 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.287127 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:03.390455 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.390381 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:03.399224 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.399199 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2"] Sep 11 19:03:03.403121 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.403101 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" Sep 11 19:03:03.405749 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.405732 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-8092ee69\"" Sep 11 19:03:03.405823 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.405803 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-8092ee69-dockercfg-4ppw8\"" Sep 11 19:03:03.416592 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.416572 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2"] Sep 11 19:03:03.438045 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.438018 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5"] Sep 11 19:03:03.441462 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.441446 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" Sep 11 19:03:03.452277 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.452252 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5"] Sep 11 19:03:03.461951 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.461925 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5120d06d-7ee2-409a-b44f-5fdad92b4334-maas-api-tls\") pod \"maas-api-e2e-derive-8092ee69-7958f477b-lcdf6\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:03.462057 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.461999 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3707f983-0133-444b-a316-bfd76a77d2e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2\" (UID: \"3707f983-0133-444b-a316-bfd76a77d2e3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" Sep 11 19:03:03.462108 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.462091 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwk7c\" (UniqueName: \"kubernetes.io/projected/3707f983-0133-444b-a316-bfd76a77d2e3-kube-api-access-fwk7c\") pod \"payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2\" (UID: \"3707f983-0133-444b-a316-bfd76a77d2e3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" Sep 11 19:03:03.464442 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.464420 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5120d06d-7ee2-409a-b44f-5fdad92b4334-maas-api-tls\") pod \"maas-api-e2e-derive-8092ee69-7958f477b-lcdf6\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:03.563080 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.563046 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5spkn\" (UniqueName: \"kubernetes.io/projected/e4674a50-4a38-401e-b059-c3f105ba0ff5-kube-api-access-5spkn\") pod \"payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5\" (UID: \"e4674a50-4a38-401e-b059-c3f105ba0ff5\") " pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" Sep 11 19:03:03.563259 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.563101 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwk7c\" (UniqueName: \"kubernetes.io/projected/3707f983-0133-444b-a316-bfd76a77d2e3-kube-api-access-fwk7c\") pod \"payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2\" (UID: \"3707f983-0133-444b-a316-bfd76a77d2e3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" Sep 11 19:03:03.563259 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.563168 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4674a50-4a38-401e-b059-c3f105ba0ff5-plugins-config-volume\") pod \"payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5\" (UID: \"e4674a50-4a38-401e-b059-c3f105ba0ff5\") " pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" Sep 11 19:03:03.563259 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.563227 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3707f983-0133-444b-a316-bfd76a77d2e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2\" (UID: \"3707f983-0133-444b-a316-bfd76a77d2e3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" Sep 11 19:03:03.563914 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.563896 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3707f983-0133-444b-a316-bfd76a77d2e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2\" (UID: \"3707f983-0133-444b-a316-bfd76a77d2e3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" Sep 11 19:03:03.570729 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.570707 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwk7c\" (UniqueName: \"kubernetes.io/projected/3707f983-0133-444b-a316-bfd76a77d2e3-kube-api-access-fwk7c\") pod \"payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2\" (UID: \"3707f983-0133-444b-a316-bfd76a77d2e3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" Sep 11 19:03:03.663663 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.663599 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5spkn\" (UniqueName: \"kubernetes.io/projected/e4674a50-4a38-401e-b059-c3f105ba0ff5-kube-api-access-5spkn\") pod \"payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5\" (UID: \"e4674a50-4a38-401e-b059-c3f105ba0ff5\") " pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" Sep 11 19:03:03.663768 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.663692 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4674a50-4a38-401e-b059-c3f105ba0ff5-plugins-config-volume\") pod \"payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5\" (UID: \"e4674a50-4a38-401e-b059-c3f105ba0ff5\") " pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" Sep 11 19:03:03.664585 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.664559 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4674a50-4a38-401e-b059-c3f105ba0ff5-plugins-config-volume\") pod \"payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5\" (UID: \"e4674a50-4a38-401e-b059-c3f105ba0ff5\") " pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" Sep 11 19:03:03.671125 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.671105 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5spkn\" (UniqueName: \"kubernetes.io/projected/e4674a50-4a38-401e-b059-c3f105ba0ff5-kube-api-access-5spkn\") pod \"payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5\" (UID: \"e4674a50-4a38-401e-b059-c3f105ba0ff5\") " pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" Sep 11 19:03:03.671267 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.671251 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:03.713970 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.713940 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" Sep 11 19:03:03.753858 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.753807 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" Sep 11 19:03:03.944659 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.944603 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6"] Sep 11 19:03:03.946910 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:03:03.946880 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5120d06d_7ee2_409a_b44f_5fdad92b4334.slice/crio-40df37038e676fedf79f2cb13799180ab94219e3e5aba8b15ddcb958617a7d38 WatchSource:0}: Error finding container 40df37038e676fedf79f2cb13799180ab94219e3e5aba8b15ddcb958617a7d38: Status 404 returned error can't find the container with id 40df37038e676fedf79f2cb13799180ab94219e3e5aba8b15ddcb958617a7d38 Sep 11 19:03:04.000014 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:03.999988 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2"] Sep 11 19:03:04.002273 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:03:04.002244 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3707f983_0133_444b_a316_bfd76a77d2e3.slice/crio-7c61b4448e0917630b59df8afd7fe85cda4bce02a20ef5826a83f74087c381dd WatchSource:0}: Error finding container 7c61b4448e0917630b59df8afd7fe85cda4bce02a20ef5826a83f74087c381dd: Status 404 returned error can't find the container with id 7c61b4448e0917630b59df8afd7fe85cda4bce02a20ef5826a83f74087c381dd Sep 11 19:03:04.039464 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:04.039442 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5"] Sep 11 19:03:04.040980 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:03:04.040955 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4674a50_4a38_401e_b059_c3f105ba0ff5.slice/crio-c2c78442d41882bcc394d6173cbd56d3af7664ac25f10ca55e65bf95021811eb WatchSource:0}: Error finding container c2c78442d41882bcc394d6173cbd56d3af7664ac25f10ca55e65bf95021811eb: Status 404 returned error can't find the container with id c2c78442d41882bcc394d6173cbd56d3af7664ac25f10ca55e65bf95021811eb Sep 11 19:03:04.478353 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:04.478316 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" event={"ID":"e4674a50-4a38-401e-b059-c3f105ba0ff5","Type":"ContainerStarted","Data":"3995f4b0b9ba6d68f1a24ff4d6254dfa2e1d7c2c3843b1192af08a7142f7e439"} Sep 11 19:03:04.478533 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:04.478413 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" Sep 11 19:03:04.478533 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:04.478440 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" event={"ID":"e4674a50-4a38-401e-b059-c3f105ba0ff5","Type":"ContainerStarted","Data":"c2c78442d41882bcc394d6173cbd56d3af7664ac25f10ca55e65bf95021811eb"} Sep 11 19:03:04.480039 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:04.480009 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" event={"ID":"3707f983-0133-444b-a316-bfd76a77d2e3","Type":"ContainerStarted","Data":"80eb79b5a6b96d2a463bea58fa9396b8388c09ce79c370c50434660db0ddc699"} Sep 11 19:03:04.480182 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:04.480053 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" event={"ID":"3707f983-0133-444b-a316-bfd76a77d2e3","Type":"ContainerStarted","Data":"7c61b4448e0917630b59df8afd7fe85cda4bce02a20ef5826a83f74087c381dd"} Sep 11 19:03:04.480182 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:04.480098 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" Sep 11 19:03:04.481593 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:04.481566 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" event={"ID":"5120d06d-7ee2-409a-b44f-5fdad92b4334","Type":"ContainerStarted","Data":"e9a170fbd0e9ea57476a14433698b27132d2afa51ed4f7cf4f1f603fdb8eb9fa"} Sep 11 19:03:04.481699 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:04.481606 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" event={"ID":"5120d06d-7ee2-409a-b44f-5fdad92b4334","Type":"ContainerStarted","Data":"40df37038e676fedf79f2cb13799180ab94219e3e5aba8b15ddcb958617a7d38"} Sep 11 19:03:04.481699 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:04.481637 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:04.494763 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:04.494722 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" podStartSLOduration=1.494708627 podStartE2EDuration="1.494708627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:03:04.492788738 +0000 UTC m=+889.946594710" watchObservedRunningTime="2026-09-11 19:03:04.494708627 +0000 UTC m=+889.948514628" Sep 11 19:03:04.509103 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:04.509061 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" podStartSLOduration=2.126234023 podStartE2EDuration="2.509047759s" totalImagesPullingTime="382.813736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:03:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:03:04.507321537 +0000 UTC m=+889.961127536" watchObservedRunningTime="2026-09-11 19:03:04.509047759 +0000 UTC m=+889.962853711" Sep 11 19:03:04.523114 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:04.523062 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" podStartSLOduration=1.523049372 podStartE2EDuration="1.523049372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:03:04.521069489 +0000 UTC m=+889.974875443" watchObservedRunningTime="2026-09-11 19:03:04.523049372 +0000 UTC m=+889.976855328" Sep 11 19:03:05.977118 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:05.977078 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:06.890914 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:06.890879 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:07.567432 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:07.567396 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:07.600804 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:07.600767 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp"] Sep 11 19:03:07.601019 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:07.600995 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" podUID="edb67128-4a54-48bd-a7cb-90efacade95e" containerName="payload-pre-processing" containerID="cri-o://3ad37b33572cb136de1cee3156bee96596aa6c3e75415e6881022d09b589f042" gracePeriod=30 Sep 11 19:03:07.667864 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:07.667830 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:07.767976 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:07.767936 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:07.774186 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:07.774145 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-89e163-fzbkj"] Sep 11 19:03:07.777899 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:07.777854 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-89e163-fzbkj"] Sep 11 19:03:07.861368 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:07.861347 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" Sep 11 19:03:07.866132 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:07.866109 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:08.009584 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.009529 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edb67128-4a54-48bd-a7cb-90efacade95e-plugins-config-volume\") pod \"edb67128-4a54-48bd-a7cb-90efacade95e\" (UID: \"edb67128-4a54-48bd-a7cb-90efacade95e\") " Sep 11 19:03:08.009749 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.009642 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6djc\" (UniqueName: \"kubernetes.io/projected/edb67128-4a54-48bd-a7cb-90efacade95e-kube-api-access-w6djc\") pod \"edb67128-4a54-48bd-a7cb-90efacade95e\" (UID: \"edb67128-4a54-48bd-a7cb-90efacade95e\") " Sep 11 19:03:08.009938 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.009917 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb67128-4a54-48bd-a7cb-90efacade95e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "edb67128-4a54-48bd-a7cb-90efacade95e" (UID: "edb67128-4a54-48bd-a7cb-90efacade95e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:03:08.011711 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.011692 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb67128-4a54-48bd-a7cb-90efacade95e-kube-api-access-w6djc" (OuterVolumeSpecName: "kube-api-access-w6djc") pod "edb67128-4a54-48bd-a7cb-90efacade95e" (UID: "edb67128-4a54-48bd-a7cb-90efacade95e"). InnerVolumeSpecName "kube-api-access-w6djc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:03:08.110424 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.110356 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edb67128-4a54-48bd-a7cb-90efacade95e-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:03:08.110424 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.110380 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6djc\" (UniqueName: \"kubernetes.io/projected/edb67128-4a54-48bd-a7cb-90efacade95e-kube-api-access-w6djc\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:03:08.187749 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.187713 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:08.284184 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.284145 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:08.506725 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.506639 2556 generic.go:356] "Generic (PLEG): container finished" podID="edb67128-4a54-48bd-a7cb-90efacade95e" containerID="3ad37b33572cb136de1cee3156bee96596aa6c3e75415e6881022d09b589f042" exitCode=0 Sep 11 19:03:08.506725 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.506718 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" event={"ID":"edb67128-4a54-48bd-a7cb-90efacade95e","Type":"ContainerDied","Data":"3ad37b33572cb136de1cee3156bee96596aa6c3e75415e6881022d09b589f042"} Sep 11 19:03:08.506965 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.506757 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" Sep 11 19:03:08.506965 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.506781 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp" event={"ID":"edb67128-4a54-48bd-a7cb-90efacade95e","Type":"ContainerDied","Data":"6f9560e835712d9fc82d610cae3305fa8a2bfdc6e0368538416d0050026c22dd"} Sep 11 19:03:08.506965 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.506802 2556 scope.go:117] "RemoveContainer" containerID="3ad37b33572cb136de1cee3156bee96596aa6c3e75415e6881022d09b589f042" Sep 11 19:03:08.516650 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.516635 2556 scope.go:117] "RemoveContainer" containerID="3ad37b33572cb136de1cee3156bee96596aa6c3e75415e6881022d09b589f042" Sep 11 19:03:08.516920 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:03:08.516901 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad37b33572cb136de1cee3156bee96596aa6c3e75415e6881022d09b589f042\": container with ID starting with 3ad37b33572cb136de1cee3156bee96596aa6c3e75415e6881022d09b589f042 not found: ID does not exist" containerID="3ad37b33572cb136de1cee3156bee96596aa6c3e75415e6881022d09b589f042" Sep 11 19:03:08.516970 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.516928 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad37b33572cb136de1cee3156bee96596aa6c3e75415e6881022d09b589f042"} err="failed to get container status \"3ad37b33572cb136de1cee3156bee96596aa6c3e75415e6881022d09b589f042\": rpc error: code = NotFound desc = could not find container \"3ad37b33572cb136de1cee3156bee96596aa6c3e75415e6881022d09b589f042\": container with ID starting with 3ad37b33572cb136de1cee3156bee96596aa6c3e75415e6881022d09b589f042 not found: ID does not exist" Sep 11 19:03:08.538312 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.538276 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp"] Sep 11 19:03:08.540180 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:08.540154 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-89e163-7f59cdb9fd-llrzp"] Sep 11 19:03:09.007669 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:09.007638 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:09.172733 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:09.172703 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00d5b5cd-db84-42a6-bb59-b52711d62ce5" path="/var/lib/kubelet/pods/00d5b5cd-db84-42a6-bb59-b52711d62ce5/volumes" Sep 11 19:03:09.173091 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:09.173078 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edb67128-4a54-48bd-a7cb-90efacade95e" path="/var/lib/kubelet/pods/edb67128-4a54-48bd-a7cb-90efacade95e/volumes" Sep 11 19:03:09.900987 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:09.900944 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:10.493051 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:10.493020 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:10.676695 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:10.676661 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:11.383414 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:11.383377 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:12.306008 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:12.305979 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:12.405903 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:12.405867 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:13.214687 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:13.214654 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:15.048184 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:15.048147 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 19:03:15.048672 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:15.048496 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 19:03:15.489800 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:15.489711 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" Sep 11 19:03:15.489961 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:15.489889 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" Sep 11 19:03:24.710659 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:24.710627 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:26.384302 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:26.384264 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:27.202184 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:27.202152 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:28.009322 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:28.009269 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:28.996678 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:28.996645 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:29.702526 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:29.702487 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:30.501386 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:30.501355 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:31.275920 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:31.275811 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:31.999930 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:31.999894 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:32.097113 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:32.097076 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:34.136454 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:34.136394 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:39.081928 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:39.081864 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:39.715198 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:39.715167 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6"] Sep 11 19:03:39.715410 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:39.715383 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" podUID="5120d06d-7ee2-409a-b44f-5fdad92b4334" containerName="maas-api" containerID="cri-o://e9a170fbd0e9ea57476a14433698b27132d2afa51ed4f7cf4f1f603fdb8eb9fa" gracePeriod=30 Sep 11 19:03:39.751012 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:39.750980 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:39.773821 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:39.773786 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5"] Sep 11 19:03:39.774064 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:39.774043 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" podUID="e4674a50-4a38-401e-b059-c3f105ba0ff5" containerName="payload-processing" containerID="cri-o://3995f4b0b9ba6d68f1a24ff4d6254dfa2e1d7c2c3843b1192af08a7142f7e439" gracePeriod=30 Sep 11 19:03:39.783843 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:39.783816 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2"] Sep 11 19:03:39.784080 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:39.784047 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" podUID="3707f983-0133-444b-a316-bfd76a77d2e3" containerName="payload-pre-processing" containerID="cri-o://80eb79b5a6b96d2a463bea58fa9396b8388c09ce79c370c50434660db0ddc699" gracePeriod=30 Sep 11 19:03:39.845949 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:39.845908 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:39.913307 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:39.913271 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:40.079433 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.079403 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" Sep 11 19:03:40.094895 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.094867 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" Sep 11 19:03:40.098699 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.098678 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:40.124476 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.124441 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:40.189420 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.189387 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:40.229763 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.229607 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbs9t\" (UniqueName: \"kubernetes.io/projected/5120d06d-7ee2-409a-b44f-5fdad92b4334-kube-api-access-fbs9t\") pod \"5120d06d-7ee2-409a-b44f-5fdad92b4334\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " Sep 11 19:03:40.229763 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.229675 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5spkn\" (UniqueName: \"kubernetes.io/projected/e4674a50-4a38-401e-b059-c3f105ba0ff5-kube-api-access-5spkn\") pod \"e4674a50-4a38-401e-b059-c3f105ba0ff5\" (UID: \"e4674a50-4a38-401e-b059-c3f105ba0ff5\") " Sep 11 19:03:40.229763 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.229742 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3707f983-0133-444b-a316-bfd76a77d2e3-plugins-config-volume\") pod \"3707f983-0133-444b-a316-bfd76a77d2e3\" (UID: \"3707f983-0133-444b-a316-bfd76a77d2e3\") " Sep 11 19:03:40.229763 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.229807 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5120d06d-7ee2-409a-b44f-5fdad92b4334-maas-api-tls\") pod \"5120d06d-7ee2-409a-b44f-5fdad92b4334\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " Sep 11 19:03:40.230183 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.229867 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5120d06d-7ee2-409a-b44f-5fdad92b4334-metrics-certs\") pod \"5120d06d-7ee2-409a-b44f-5fdad92b4334\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " Sep 11 19:03:40.230183 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.229929 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4674a50-4a38-401e-b059-c3f105ba0ff5-plugins-config-volume\") pod \"e4674a50-4a38-401e-b059-c3f105ba0ff5\" (UID: \"e4674a50-4a38-401e-b059-c3f105ba0ff5\") " Sep 11 19:03:40.230183 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.229968 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5120d06d-7ee2-409a-b44f-5fdad92b4334-openshift-service-ca\") pod \"5120d06d-7ee2-409a-b44f-5fdad92b4334\" (UID: \"5120d06d-7ee2-409a-b44f-5fdad92b4334\") " Sep 11 19:03:40.230183 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.230014 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwk7c\" (UniqueName: \"kubernetes.io/projected/3707f983-0133-444b-a316-bfd76a77d2e3-kube-api-access-fwk7c\") pod \"3707f983-0133-444b-a316-bfd76a77d2e3\" (UID: \"3707f983-0133-444b-a316-bfd76a77d2e3\") " Sep 11 19:03:40.231817 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.231093 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3707f983-0133-444b-a316-bfd76a77d2e3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3707f983-0133-444b-a316-bfd76a77d2e3" (UID: "3707f983-0133-444b-a316-bfd76a77d2e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:03:40.231817 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.231652 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5120d06d-7ee2-409a-b44f-5fdad92b4334-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5120d06d-7ee2-409a-b44f-5fdad92b4334" (UID: "5120d06d-7ee2-409a-b44f-5fdad92b4334"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:03:40.232362 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.232312 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4674a50-4a38-401e-b059-c3f105ba0ff5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4674a50-4a38-401e-b059-c3f105ba0ff5" (UID: "e4674a50-4a38-401e-b059-c3f105ba0ff5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:03:40.233637 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.233607 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5120d06d-7ee2-409a-b44f-5fdad92b4334-kube-api-access-fbs9t" (OuterVolumeSpecName: "kube-api-access-fbs9t") pod "5120d06d-7ee2-409a-b44f-5fdad92b4334" (UID: "5120d06d-7ee2-409a-b44f-5fdad92b4334"). InnerVolumeSpecName "kube-api-access-fbs9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:03:40.234015 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.233983 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3707f983-0133-444b-a316-bfd76a77d2e3-kube-api-access-fwk7c" (OuterVolumeSpecName: "kube-api-access-fwk7c") pod "3707f983-0133-444b-a316-bfd76a77d2e3" (UID: "3707f983-0133-444b-a316-bfd76a77d2e3"). InnerVolumeSpecName "kube-api-access-fwk7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:03:40.235958 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.234953 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5120d06d-7ee2-409a-b44f-5fdad92b4334-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5120d06d-7ee2-409a-b44f-5fdad92b4334" (UID: "5120d06d-7ee2-409a-b44f-5fdad92b4334"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:03:40.235958 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.235447 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5120d06d-7ee2-409a-b44f-5fdad92b4334-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5120d06d-7ee2-409a-b44f-5fdad92b4334" (UID: "5120d06d-7ee2-409a-b44f-5fdad92b4334"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:03:40.235958 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.235463 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4674a50-4a38-401e-b059-c3f105ba0ff5-kube-api-access-5spkn" (OuterVolumeSpecName: "kube-api-access-5spkn") pod "e4674a50-4a38-401e-b059-c3f105ba0ff5" (UID: "e4674a50-4a38-401e-b059-c3f105ba0ff5"). InnerVolumeSpecName "kube-api-access-5spkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:03:40.331426 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.331385 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwk7c\" (UniqueName: \"kubernetes.io/projected/3707f983-0133-444b-a316-bfd76a77d2e3-kube-api-access-fwk7c\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:03:40.331426 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.331427 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbs9t\" (UniqueName: \"kubernetes.io/projected/5120d06d-7ee2-409a-b44f-5fdad92b4334-kube-api-access-fbs9t\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:03:40.331665 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.331442 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5spkn\" (UniqueName: \"kubernetes.io/projected/e4674a50-4a38-401e-b059-c3f105ba0ff5-kube-api-access-5spkn\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:03:40.331665 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.331456 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3707f983-0133-444b-a316-bfd76a77d2e3-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:03:40.331665 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.331473 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5120d06d-7ee2-409a-b44f-5fdad92b4334-maas-api-tls\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:03:40.331665 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.331487 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5120d06d-7ee2-409a-b44f-5fdad92b4334-metrics-certs\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:03:40.331665 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.331500 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4674a50-4a38-401e-b059-c3f105ba0ff5-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:03:40.331665 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.331515 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5120d06d-7ee2-409a-b44f-5fdad92b4334-openshift-service-ca\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:03:40.341408 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.341382 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:40.691699 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.691604 2556 generic.go:356] "Generic (PLEG): container finished" podID="e4674a50-4a38-401e-b059-c3f105ba0ff5" containerID="3995f4b0b9ba6d68f1a24ff4d6254dfa2e1d7c2c3843b1192af08a7142f7e439" exitCode=0 Sep 11 19:03:40.691699 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.691651 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" Sep 11 19:03:40.691944 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.691697 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" event={"ID":"e4674a50-4a38-401e-b059-c3f105ba0ff5","Type":"ContainerDied","Data":"3995f4b0b9ba6d68f1a24ff4d6254dfa2e1d7c2c3843b1192af08a7142f7e439"} Sep 11 19:03:40.691944 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.691763 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5" event={"ID":"e4674a50-4a38-401e-b059-c3f105ba0ff5","Type":"ContainerDied","Data":"c2c78442d41882bcc394d6173cbd56d3af7664ac25f10ca55e65bf95021811eb"} Sep 11 19:03:40.691944 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.691793 2556 scope.go:117] "RemoveContainer" containerID="3995f4b0b9ba6d68f1a24ff4d6254dfa2e1d7c2c3843b1192af08a7142f7e439" Sep 11 19:03:40.693138 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.693105 2556 generic.go:356] "Generic (PLEG): container finished" podID="3707f983-0133-444b-a316-bfd76a77d2e3" containerID="80eb79b5a6b96d2a463bea58fa9396b8388c09ce79c370c50434660db0ddc699" exitCode=0 Sep 11 19:03:40.693255 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.693144 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" Sep 11 19:03:40.693255 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.693160 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" event={"ID":"3707f983-0133-444b-a316-bfd76a77d2e3","Type":"ContainerDied","Data":"80eb79b5a6b96d2a463bea58fa9396b8388c09ce79c370c50434660db0ddc699"} Sep 11 19:03:40.693255 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.693209 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2" event={"ID":"3707f983-0133-444b-a316-bfd76a77d2e3","Type":"ContainerDied","Data":"7c61b4448e0917630b59df8afd7fe85cda4bce02a20ef5826a83f74087c381dd"} Sep 11 19:03:40.694746 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.694725 2556 generic.go:356] "Generic (PLEG): container finished" podID="5120d06d-7ee2-409a-b44f-5fdad92b4334" containerID="e9a170fbd0e9ea57476a14433698b27132d2afa51ed4f7cf4f1f603fdb8eb9fa" exitCode=0 Sep 11 19:03:40.694862 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.694767 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" Sep 11 19:03:40.694862 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.694788 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" event={"ID":"5120d06d-7ee2-409a-b44f-5fdad92b4334","Type":"ContainerDied","Data":"e9a170fbd0e9ea57476a14433698b27132d2afa51ed4f7cf4f1f603fdb8eb9fa"} Sep 11 19:03:40.694862 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.694829 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6" event={"ID":"5120d06d-7ee2-409a-b44f-5fdad92b4334","Type":"ContainerDied","Data":"40df37038e676fedf79f2cb13799180ab94219e3e5aba8b15ddcb958617a7d38"} Sep 11 19:03:40.708243 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.708217 2556 scope.go:117] "RemoveContainer" containerID="3995f4b0b9ba6d68f1a24ff4d6254dfa2e1d7c2c3843b1192af08a7142f7e439" Sep 11 19:03:40.708633 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:03:40.708607 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3995f4b0b9ba6d68f1a24ff4d6254dfa2e1d7c2c3843b1192af08a7142f7e439\": container with ID starting with 3995f4b0b9ba6d68f1a24ff4d6254dfa2e1d7c2c3843b1192af08a7142f7e439 not found: ID does not exist" containerID="3995f4b0b9ba6d68f1a24ff4d6254dfa2e1d7c2c3843b1192af08a7142f7e439" Sep 11 19:03:40.708743 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.708641 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3995f4b0b9ba6d68f1a24ff4d6254dfa2e1d7c2c3843b1192af08a7142f7e439"} err="failed to get container status \"3995f4b0b9ba6d68f1a24ff4d6254dfa2e1d7c2c3843b1192af08a7142f7e439\": rpc error: code = NotFound desc = could not find container \"3995f4b0b9ba6d68f1a24ff4d6254dfa2e1d7c2c3843b1192af08a7142f7e439\": container with ID starting with 3995f4b0b9ba6d68f1a24ff4d6254dfa2e1d7c2c3843b1192af08a7142f7e439 not found: ID does not exist" Sep 11 19:03:40.708743 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.708659 2556 scope.go:117] "RemoveContainer" containerID="80eb79b5a6b96d2a463bea58fa9396b8388c09ce79c370c50434660db0ddc699" Sep 11 19:03:40.718367 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.718189 2556 scope.go:117] "RemoveContainer" containerID="80eb79b5a6b96d2a463bea58fa9396b8388c09ce79c370c50434660db0ddc699" Sep 11 19:03:40.718480 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:03:40.718461 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80eb79b5a6b96d2a463bea58fa9396b8388c09ce79c370c50434660db0ddc699\": container with ID starting with 80eb79b5a6b96d2a463bea58fa9396b8388c09ce79c370c50434660db0ddc699 not found: ID does not exist" containerID="80eb79b5a6b96d2a463bea58fa9396b8388c09ce79c370c50434660db0ddc699" Sep 11 19:03:40.718562 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.718492 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80eb79b5a6b96d2a463bea58fa9396b8388c09ce79c370c50434660db0ddc699"} err="failed to get container status \"80eb79b5a6b96d2a463bea58fa9396b8388c09ce79c370c50434660db0ddc699\": rpc error: code = NotFound desc = could not find container \"80eb79b5a6b96d2a463bea58fa9396b8388c09ce79c370c50434660db0ddc699\": container with ID starting with 80eb79b5a6b96d2a463bea58fa9396b8388c09ce79c370c50434660db0ddc699 not found: ID does not exist" Sep 11 19:03:40.718562 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.718515 2556 scope.go:117] "RemoveContainer" containerID="e9a170fbd0e9ea57476a14433698b27132d2afa51ed4f7cf4f1f603fdb8eb9fa" Sep 11 19:03:40.726275 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.726245 2556 scope.go:117] "RemoveContainer" containerID="e9a170fbd0e9ea57476a14433698b27132d2afa51ed4f7cf4f1f603fdb8eb9fa" Sep 11 19:03:40.726581 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:03:40.726554 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a170fbd0e9ea57476a14433698b27132d2afa51ed4f7cf4f1f603fdb8eb9fa\": container with ID starting with e9a170fbd0e9ea57476a14433698b27132d2afa51ed4f7cf4f1f603fdb8eb9fa not found: ID does not exist" containerID="e9a170fbd0e9ea57476a14433698b27132d2afa51ed4f7cf4f1f603fdb8eb9fa" Sep 11 19:03:40.726713 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.726651 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a170fbd0e9ea57476a14433698b27132d2afa51ed4f7cf4f1f603fdb8eb9fa"} err="failed to get container status \"e9a170fbd0e9ea57476a14433698b27132d2afa51ed4f7cf4f1f603fdb8eb9fa\": rpc error: code = NotFound desc = could not find container \"e9a170fbd0e9ea57476a14433698b27132d2afa51ed4f7cf4f1f603fdb8eb9fa\": container with ID starting with e9a170fbd0e9ea57476a14433698b27132d2afa51ed4f7cf4f1f603fdb8eb9fa not found: ID does not exist" Sep 11 19:03:40.729090 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.729070 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5"] Sep 11 19:03:40.733506 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.733486 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8092ee69-67cdbf7cb9-69wv5"] Sep 11 19:03:40.741407 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.741382 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2"] Sep 11 19:03:40.745215 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.745192 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8092ee69-5bc7bb9cbb-bvlg2"] Sep 11 19:03:40.755443 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.755415 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6"] Sep 11 19:03:40.758772 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.758751 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8092ee69-7958f477b-lcdf6"] Sep 11 19:03:40.833368 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.833314 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:40.999014 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:40.998935 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:41.097289 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:41.097259 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:41.171763 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:41.171736 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3707f983-0133-444b-a316-bfd76a77d2e3" path="/var/lib/kubelet/pods/3707f983-0133-444b-a316-bfd76a77d2e3/volumes" Sep 11 19:03:41.172141 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:41.172129 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5120d06d-7ee2-409a-b44f-5fdad92b4334" path="/var/lib/kubelet/pods/5120d06d-7ee2-409a-b44f-5fdad92b4334/volumes" Sep 11 19:03:41.172447 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:41.172436 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4674a50-4a38-401e-b059-c3f105ba0ff5" path="/var/lib/kubelet/pods/e4674a50-4a38-401e-b059-c3f105ba0ff5/volumes" Sep 11 19:03:41.198320 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:41.198291 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:41.923872 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:41.923836 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:42.479313 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:42.479279 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:42.778464 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:42.778386 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:43.544454 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:43.544424 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:44.443754 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:44.443723 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:48.354703 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:48.354668 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:48.457356 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:48.457319 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:48.555867 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:48.555828 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:48.655750 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:48.655654 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:49.040872 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:49.040840 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:50.090508 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:50.090471 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:50.373928 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:50.373836 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:50.797555 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:50.797508 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:51.083480 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:51.083390 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:51.449752 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:51.449651 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:54.051807 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:54.051774 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:54.219654 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:54.219601 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:54.304407 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:54.304319 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:55.112592 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:55.112560 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:55.322307 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:55.322259 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:55.954891 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:55.954852 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:56.843607 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:56.843573 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:57.531309 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.531258 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb"] Sep 11 19:03:57.531862 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.531839 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3707f983-0133-444b-a316-bfd76a77d2e3" containerName="payload-pre-processing" Sep 11 19:03:57.532023 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.532007 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3707f983-0133-444b-a316-bfd76a77d2e3" containerName="payload-pre-processing" Sep 11 19:03:57.532130 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.532117 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4674a50-4a38-401e-b059-c3f105ba0ff5" containerName="payload-processing" Sep 11 19:03:57.532223 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.532211 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4674a50-4a38-401e-b059-c3f105ba0ff5" containerName="payload-processing" Sep 11 19:03:57.532323 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.532312 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5120d06d-7ee2-409a-b44f-5fdad92b4334" containerName="maas-api" Sep 11 19:03:57.532406 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.532396 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5120d06d-7ee2-409a-b44f-5fdad92b4334" containerName="maas-api" Sep 11 19:03:57.532485 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.532469 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb67128-4a54-48bd-a7cb-90efacade95e" containerName="payload-pre-processing" Sep 11 19:03:57.532580 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.532566 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb67128-4a54-48bd-a7cb-90efacade95e" containerName="payload-pre-processing" Sep 11 19:03:57.532733 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.532720 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5120d06d-7ee2-409a-b44f-5fdad92b4334" containerName="maas-api" Sep 11 19:03:57.532828 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.532817 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb67128-4a54-48bd-a7cb-90efacade95e" containerName="payload-pre-processing" Sep 11 19:03:57.532907 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.532898 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3707f983-0133-444b-a316-bfd76a77d2e3" containerName="payload-pre-processing" Sep 11 19:03:57.532979 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.532971 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4674a50-4a38-401e-b059-c3f105ba0ff5" containerName="payload-processing" Sep 11 19:03:57.537800 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.537773 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:03:57.541638 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.541609 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-f60615-serving-cert\"" Sep 11 19:03:57.541975 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.541879 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 19:03:57.542153 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.542137 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6bftc\"" Sep 11 19:03:57.551276 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.551147 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb"] Sep 11 19:03:57.570432 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.570395 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:57.636687 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.636649 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:57.698974 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.698919 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7gx5\" (UniqueName: \"kubernetes.io/projected/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-kube-api-access-g7gx5\") pod \"maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:03:57.699174 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.699020 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-openshift-service-ca\") pod \"maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:03:57.699174 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.699143 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-maas-api-tls\") pod \"maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:03:57.699269 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.699177 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-metrics-certs\") pod \"maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:03:57.799748 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.799709 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-metrics-certs\") pod \"maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:03:57.799964 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.799794 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7gx5\" (UniqueName: \"kubernetes.io/projected/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-kube-api-access-g7gx5\") pod \"maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:03:57.799964 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.799851 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-openshift-service-ca\") pod \"maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:03:57.799964 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.799918 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-maas-api-tls\") pod \"maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:03:57.800139 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:03:57.800008 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-serving-cert: secret "maas-api-e2e-explicit-f60615-serving-cert" not found Sep 11 19:03:57.800139 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:03:57.800075 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-maas-api-tls podName:c48b656e-477e-40c4-85c5-7f1e7ea6f2e6 nodeName:}" failed. No retries permitted until 2026-09-11 19:03:58.300055676 +0000 UTC m=+943.753861609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-maas-api-tls") pod "maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" (UID: "c48b656e-477e-40c4-85c5-7f1e7ea6f2e6") : secret "maas-api-e2e-explicit-f60615-serving-cert" not found Sep 11 19:03:57.800608 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.800587 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-openshift-service-ca\") pod \"maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:03:57.802195 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.802165 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-metrics-certs\") pod \"maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:03:57.808789 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.808763 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7gx5\" (UniqueName: \"kubernetes.io/projected/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-kube-api-access-g7gx5\") pod \"maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:03:57.933214 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:57.933182 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:58.067968 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.067867 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:58.165682 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.165647 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:03:58.187291 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.187250 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k"] Sep 11 19:03:58.191283 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.191263 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" Sep 11 19:03:58.193493 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.193468 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-f60615-dockercfg-8hlvp\"" Sep 11 19:03:58.193749 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.193730 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-f60615\"" Sep 11 19:03:58.203044 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.203013 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1198a82-8895-4c64-9569-4dfc4cf33f28-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f60615-57548b9865-lnq8k\" (UID: \"d1198a82-8895-4c64-9569-4dfc4cf33f28\") " pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" Sep 11 19:03:58.203217 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.203116 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv9fw\" (UniqueName: \"kubernetes.io/projected/d1198a82-8895-4c64-9569-4dfc4cf33f28-kube-api-access-pv9fw\") pod \"payload-processing-e2e-explicit-f60615-57548b9865-lnq8k\" (UID: \"d1198a82-8895-4c64-9569-4dfc4cf33f28\") " pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" Sep 11 19:03:58.204276 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.204251 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k"] Sep 11 19:03:58.252686 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.252647 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv"] Sep 11 19:03:58.256201 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.256183 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" Sep 11 19:03:58.265507 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.265478 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv"] Sep 11 19:03:58.304633 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.304518 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-maas-api-tls\") pod \"maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:03:58.304881 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.304637 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1198a82-8895-4c64-9569-4dfc4cf33f28-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f60615-57548b9865-lnq8k\" (UID: \"d1198a82-8895-4c64-9569-4dfc4cf33f28\") " pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" Sep 11 19:03:58.304881 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.304732 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv9fw\" (UniqueName: \"kubernetes.io/projected/d1198a82-8895-4c64-9569-4dfc4cf33f28-kube-api-access-pv9fw\") pod \"payload-processing-e2e-explicit-f60615-57548b9865-lnq8k\" (UID: \"d1198a82-8895-4c64-9569-4dfc4cf33f28\") " pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" Sep 11 19:03:58.305704 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.305674 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1198a82-8895-4c64-9569-4dfc4cf33f28-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f60615-57548b9865-lnq8k\" (UID: \"d1198a82-8895-4c64-9569-4dfc4cf33f28\") " pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" Sep 11 19:03:58.307852 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.307824 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-maas-api-tls\") pod \"maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:03:58.313185 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.313155 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv9fw\" (UniqueName: \"kubernetes.io/projected/d1198a82-8895-4c64-9569-4dfc4cf33f28-kube-api-access-pv9fw\") pod \"payload-processing-e2e-explicit-f60615-57548b9865-lnq8k\" (UID: \"d1198a82-8895-4c64-9569-4dfc4cf33f28\") " pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" Sep 11 19:03:58.405440 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.405346 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knlpn\" (UniqueName: \"kubernetes.io/projected/d25897cb-64ad-4470-a8fe-b25d65436112-kube-api-access-knlpn\") pod \"payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv\" (UID: \"d25897cb-64ad-4470-a8fe-b25d65436112\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" Sep 11 19:03:58.405440 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.405416 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25897cb-64ad-4470-a8fe-b25d65436112-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv\" (UID: \"d25897cb-64ad-4470-a8fe-b25d65436112\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" Sep 11 19:03:58.470477 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.470419 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:03:58.503760 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.503722 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" Sep 11 19:03:58.506132 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.506101 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knlpn\" (UniqueName: \"kubernetes.io/projected/d25897cb-64ad-4470-a8fe-b25d65436112-kube-api-access-knlpn\") pod \"payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv\" (UID: \"d25897cb-64ad-4470-a8fe-b25d65436112\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" Sep 11 19:03:58.506248 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.506164 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25897cb-64ad-4470-a8fe-b25d65436112-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv\" (UID: \"d25897cb-64ad-4470-a8fe-b25d65436112\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" Sep 11 19:03:58.506918 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.506894 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25897cb-64ad-4470-a8fe-b25d65436112-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv\" (UID: \"d25897cb-64ad-4470-a8fe-b25d65436112\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" Sep 11 19:03:58.514756 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.514726 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knlpn\" (UniqueName: \"kubernetes.io/projected/d25897cb-64ad-4470-a8fe-b25d65436112-kube-api-access-knlpn\") pod \"payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv\" (UID: \"d25897cb-64ad-4470-a8fe-b25d65436112\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" Sep 11 19:03:58.568805 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.568691 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" Sep 11 19:03:58.762276 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.762249 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb"] Sep 11 19:03:58.764166 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:03:58.764137 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48b656e_477e_40c4_85c5_7f1e7ea6f2e6.slice/crio-c5d2f75b5101f1c8374c4974f58910144af7d4b553f6a3fef8db5bc2ab639c07 WatchSource:0}: Error finding container c5d2f75b5101f1c8374c4974f58910144af7d4b553f6a3fef8db5bc2ab639c07: Status 404 returned error can't find the container with id c5d2f75b5101f1c8374c4974f58910144af7d4b553f6a3fef8db5bc2ab639c07 Sep 11 19:03:58.765525 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.765508 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 19:03:58.805148 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.805113 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" event={"ID":"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6","Type":"ContainerStarted","Data":"c5d2f75b5101f1c8374c4974f58910144af7d4b553f6a3fef8db5bc2ab639c07"} Sep 11 19:03:58.809631 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.809602 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k"] Sep 11 19:03:58.810533 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:03:58.810508 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1198a82_8895_4c64_9569_4dfc4cf33f28.slice/crio-bb3999c04ce1ff152d735b255c6649c80c9e71cde80e1810223e09f1664db1aa WatchSource:0}: Error finding container bb3999c04ce1ff152d735b255c6649c80c9e71cde80e1810223e09f1664db1aa: Status 404 returned error can't find the container with id bb3999c04ce1ff152d735b255c6649c80c9e71cde80e1810223e09f1664db1aa Sep 11 19:03:58.855421 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:58.855398 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv"] Sep 11 19:03:58.858613 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:03:58.858585 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25897cb_64ad_4470_a8fe_b25d65436112.slice/crio-5dca09e1fcfd35526fb355a71ecc5d9284d832ef10fbf5291d25042f52b090b7 WatchSource:0}: Error finding container 5dca09e1fcfd35526fb355a71ecc5d9284d832ef10fbf5291d25042f52b090b7: Status 404 returned error can't find the container with id 5dca09e1fcfd35526fb355a71ecc5d9284d832ef10fbf5291d25042f52b090b7 Sep 11 19:03:59.812728 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:59.812692 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" event={"ID":"d25897cb-64ad-4470-a8fe-b25d65436112","Type":"ContainerStarted","Data":"c9e5f3c696b0de2d114f7cbf58364e217963facc316fc274b8eddf53a558afee"} Sep 11 19:03:59.813174 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:59.812760 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" Sep 11 19:03:59.813174 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:59.812786 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" event={"ID":"d25897cb-64ad-4470-a8fe-b25d65436112","Type":"ContainerStarted","Data":"5dca09e1fcfd35526fb355a71ecc5d9284d832ef10fbf5291d25042f52b090b7"} Sep 11 19:03:59.814314 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:59.814285 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" event={"ID":"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6","Type":"ContainerStarted","Data":"0f9f59581f05da4a82dfc6ef90c0e9b50082bddb40e5f93b69d5b9262b2be18d"} Sep 11 19:03:59.814439 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:59.814341 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:03:59.815831 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:59.815795 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" event={"ID":"d1198a82-8895-4c64-9569-4dfc4cf33f28","Type":"ContainerStarted","Data":"5913c54128a5571637365853f192ade1759f58257c3e3a975fd137ca78bb06bc"} Sep 11 19:03:59.815956 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:59.815847 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" event={"ID":"d1198a82-8895-4c64-9569-4dfc4cf33f28","Type":"ContainerStarted","Data":"bb3999c04ce1ff152d735b255c6649c80c9e71cde80e1810223e09f1664db1aa"} Sep 11 19:03:59.815956 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:59.815915 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" Sep 11 19:03:59.828963 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:59.828909 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" podStartSLOduration=1.828894469 podStartE2EDuration="1.828894469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:03:59.827094423 +0000 UTC m=+945.280900389" watchObservedRunningTime="2026-09-11 19:03:59.828894469 +0000 UTC m=+945.282700504" Sep 11 19:03:59.843889 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:59.843837 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" podStartSLOduration=2.436322172 podStartE2EDuration="2.843819575s" totalImagesPullingTime="407.497403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:03:59.841772 +0000 UTC m=+945.295577979" watchObservedRunningTime="2026-09-11 19:03:59.843819575 +0000 UTC m=+945.297625584" Sep 11 19:03:59.858403 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:03:59.858345 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" podStartSLOduration=1.858328612 podStartE2EDuration="1.858328612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:03:59.856052374 +0000 UTC m=+945.309858346" watchObservedRunningTime="2026-09-11 19:03:59.858328612 +0000 UTC m=+945.312134567" Sep 11 19:04:05.827523 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:05.827493 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:04:10.823337 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:10.823301 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" Sep 11 19:04:10.823827 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:10.823535 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" Sep 11 19:04:14.951843 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:14.951788 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:19.623836 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:19.623793 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:21.021913 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:21.021877 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:21.125473 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:21.125441 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:21.222218 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:21.222186 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:21.331742 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:21.331707 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:22.263387 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:22.263345 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:22.958872 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:22.958838 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:23.656103 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:23.656066 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:24.231910 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:24.231877 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:24.481040 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:24.481007 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:25.192504 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:25.192473 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:37.041516 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:37.041480 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:42.950128 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:42.950097 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:43.053976 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:43.053942 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:43.150289 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:43.150251 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:53.604426 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:53.604394 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:04:54.360618 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:04:54.360583 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:00.278632 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:00.278596 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:00.382185 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:00.382152 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:00.486769 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:00.486733 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:00.592992 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:00.592959 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:01.180871 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:01.180840 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:01.420129 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:01.420100 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:01.526140 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:01.526056 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:02.822655 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:02.822621 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:08.765630 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:08.765590 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:10.153798 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:10.153758 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:11.212925 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:11.212891 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:11.313302 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:11.313273 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:15.473736 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:15.473697 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:15.574450 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:15.574419 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:15.699098 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:15.699065 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:16.130057 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:16.130023 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:16.662967 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:16.662938 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:20.435183 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:20.435151 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:21.506969 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:21.506935 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:21.605916 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:21.605887 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:21.939334 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:21.939301 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:22.225076 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:22.224991 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:23.106697 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:23.106660 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:23.987267 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:23.987225 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:24.837580 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:24.837531 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:25.534615 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.534583 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh"] Sep 11 19:05:25.538457 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.538434 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:25.540525 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.540500 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9a60dde0-serving-cert\"" Sep 11 19:05:25.547896 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.547651 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh"] Sep 11 19:05:25.566111 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.566087 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:25.621380 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.621353 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpw2p\" (UniqueName: \"kubernetes.io/projected/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-kube-api-access-fpw2p\") pod \"maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:25.621488 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.621396 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-maas-api-tls\") pod \"maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:25.621488 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.621474 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-openshift-service-ca\") pod \"maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:25.621610 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.621491 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-metrics-certs\") pod \"maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:25.628245 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.628222 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:25.722569 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.722507 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-openshift-service-ca\") pod \"maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:25.722768 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.722575 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-metrics-certs\") pod \"maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:25.722768 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.722652 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpw2p\" (UniqueName: \"kubernetes.io/projected/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-kube-api-access-fpw2p\") pod \"maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:25.722768 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.722710 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-maas-api-tls\") pod \"maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:25.722926 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:05:25.722841 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-serving-cert: secret "maas-api-e2e-mt-9a60dde0-serving-cert" not found Sep 11 19:05:25.722926 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:05:25.722923 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-maas-api-tls podName:c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af nodeName:}" failed. No retries permitted until 2026-09-11 19:05:26.222900198 +0000 UTC m=+1031.676706133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-maas-api-tls") pod "maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" (UID: "c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af") : secret "maas-api-e2e-mt-9a60dde0-serving-cert" not found Sep 11 19:05:25.723245 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.723228 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-openshift-service-ca\") pod \"maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:25.725095 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.725078 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-metrics-certs\") pod \"maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:25.733498 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.733472 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpw2p\" (UniqueName: \"kubernetes.io/projected/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-kube-api-access-fpw2p\") pod \"maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:25.905318 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:25.905234 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:26.124608 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.124577 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:26.219723 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.219620 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:26.226885 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.226858 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-maas-api-tls\") pod \"maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:26.229250 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.229234 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-maas-api-tls\") pod \"maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:26.262558 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.259087 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj"] Sep 11 19:05:26.264951 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.264930 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" Sep 11 19:05:26.267076 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.267053 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9a60dde0\"" Sep 11 19:05:26.267178 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.267053 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9a60dde0-dockercfg-2f7l4\"" Sep 11 19:05:26.271741 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.271704 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj"] Sep 11 19:05:26.295838 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.295815 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285"] Sep 11 19:05:26.299709 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.299691 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" Sep 11 19:05:26.308834 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.308808 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285"] Sep 11 19:05:26.327520 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.327491 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d500be3c-74c8-4e0d-9092-b93f0069a40d-plugins-config-volume\") pod \"payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj\" (UID: \"d500be3c-74c8-4e0d-9092-b93f0069a40d\") " pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" Sep 11 19:05:26.327692 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.327664 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b401684-b363-474a-8834-29dd7a6357c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285\" (UID: \"9b401684-b363-474a-8834-29dd7a6357c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" Sep 11 19:05:26.327814 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.327727 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9ht2\" (UniqueName: \"kubernetes.io/projected/d500be3c-74c8-4e0d-9092-b93f0069a40d-kube-api-access-l9ht2\") pod \"payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj\" (UID: \"d500be3c-74c8-4e0d-9092-b93f0069a40d\") " pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" Sep 11 19:05:26.327814 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.327752 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hhj5\" (UniqueName: \"kubernetes.io/projected/9b401684-b363-474a-8834-29dd7a6357c8-kube-api-access-2hhj5\") pod \"payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285\" (UID: \"9b401684-b363-474a-8834-29dd7a6357c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" Sep 11 19:05:26.429054 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.429026 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d500be3c-74c8-4e0d-9092-b93f0069a40d-plugins-config-volume\") pod \"payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj\" (UID: \"d500be3c-74c8-4e0d-9092-b93f0069a40d\") " pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" Sep 11 19:05:26.429221 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.429091 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b401684-b363-474a-8834-29dd7a6357c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285\" (UID: \"9b401684-b363-474a-8834-29dd7a6357c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" Sep 11 19:05:26.429221 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.429136 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9ht2\" (UniqueName: \"kubernetes.io/projected/d500be3c-74c8-4e0d-9092-b93f0069a40d-kube-api-access-l9ht2\") pod \"payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj\" (UID: \"d500be3c-74c8-4e0d-9092-b93f0069a40d\") " pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" Sep 11 19:05:26.429221 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.429160 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hhj5\" (UniqueName: \"kubernetes.io/projected/9b401684-b363-474a-8834-29dd7a6357c8-kube-api-access-2hhj5\") pod \"payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285\" (UID: \"9b401684-b363-474a-8834-29dd7a6357c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" Sep 11 19:05:26.429892 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.429873 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d500be3c-74c8-4e0d-9092-b93f0069a40d-plugins-config-volume\") pod \"payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj\" (UID: \"d500be3c-74c8-4e0d-9092-b93f0069a40d\") " pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" Sep 11 19:05:26.429938 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.429872 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b401684-b363-474a-8834-29dd7a6357c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285\" (UID: \"9b401684-b363-474a-8834-29dd7a6357c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" Sep 11 19:05:26.436504 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.436481 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hhj5\" (UniqueName: \"kubernetes.io/projected/9b401684-b363-474a-8834-29dd7a6357c8-kube-api-access-2hhj5\") pod \"payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285\" (UID: \"9b401684-b363-474a-8834-29dd7a6357c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" Sep 11 19:05:26.436634 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.436562 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9ht2\" (UniqueName: \"kubernetes.io/projected/d500be3c-74c8-4e0d-9092-b93f0069a40d-kube-api-access-l9ht2\") pod \"payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj\" (UID: \"d500be3c-74c8-4e0d-9092-b93f0069a40d\") " pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" Sep 11 19:05:26.452399 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.452361 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:26.578621 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.578589 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" Sep 11 19:05:26.610555 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.610521 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" Sep 11 19:05:26.727497 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.727475 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh"] Sep 11 19:05:26.729581 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:05:26.729528 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9cdfc5e_bcbb_4f79_9fc3_6a58f388b0af.slice/crio-fc5b9ca0cf8ff31e97d9047c8047327ae393b7d3d3d44525d90f9a9aeeb240ec WatchSource:0}: Error finding container fc5b9ca0cf8ff31e97d9047c8047327ae393b7d3d3d44525d90f9a9aeeb240ec: Status 404 returned error can't find the container with id fc5b9ca0cf8ff31e97d9047c8047327ae393b7d3d3d44525d90f9a9aeeb240ec Sep 11 19:05:26.855401 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.855361 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj"] Sep 11 19:05:26.858024 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:05:26.857995 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd500be3c_74c8_4e0d_9092_b93f0069a40d.slice/crio-b5a211cc48de114bfc3b18479911ea51686e3a320a4008c390dca6d732b0d80c WatchSource:0}: Error finding container b5a211cc48de114bfc3b18479911ea51686e3a320a4008c390dca6d732b0d80c: Status 404 returned error can't find the container with id b5a211cc48de114bfc3b18479911ea51686e3a320a4008c390dca6d732b0d80c Sep 11 19:05:26.893880 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:26.893858 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285"] Sep 11 19:05:26.897700 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:05:26.897676 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b401684_b363_474a_8834_29dd7a6357c8.slice/crio-bf8bbdb8bbf72a30585044bd331ad571b9625c6741c81f03467b2293557ec947 WatchSource:0}: Error finding container bf8bbdb8bbf72a30585044bd331ad571b9625c6741c81f03467b2293557ec947: Status 404 returned error can't find the container with id bf8bbdb8bbf72a30585044bd331ad571b9625c6741c81f03467b2293557ec947 Sep 11 19:05:27.303364 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:27.302466 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" event={"ID":"9b401684-b363-474a-8834-29dd7a6357c8","Type":"ContainerStarted","Data":"b7dd9c66372c8f3bd8624550251eeb2c0d36974ed2db9ceb8b028f16c76a4f86"} Sep 11 19:05:27.303364 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:27.302530 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" event={"ID":"9b401684-b363-474a-8834-29dd7a6357c8","Type":"ContainerStarted","Data":"bf8bbdb8bbf72a30585044bd331ad571b9625c6741c81f03467b2293557ec947"} Sep 11 19:05:27.303364 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:27.303311 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" Sep 11 19:05:27.307310 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:27.306010 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" event={"ID":"d500be3c-74c8-4e0d-9092-b93f0069a40d","Type":"ContainerStarted","Data":"d72463236024a05e0d811861bafcf0993a2c2ff1234aa9819de9fb56f99422fe"} Sep 11 19:05:27.307310 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:27.306058 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" event={"ID":"d500be3c-74c8-4e0d-9092-b93f0069a40d","Type":"ContainerStarted","Data":"b5a211cc48de114bfc3b18479911ea51686e3a320a4008c390dca6d732b0d80c"} Sep 11 19:05:27.307310 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:27.306569 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" Sep 11 19:05:27.314078 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:27.313554 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" event={"ID":"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af","Type":"ContainerStarted","Data":"5ea14a4134184d5c011e8244e48f39b9c1dccf6cf0954da7548e6ef85ecbeaee"} Sep 11 19:05:27.314078 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:27.313597 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" event={"ID":"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af","Type":"ContainerStarted","Data":"fc5b9ca0cf8ff31e97d9047c8047327ae393b7d3d3d44525d90f9a9aeeb240ec"} Sep 11 19:05:27.314078 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:27.314033 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:27.324557 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:27.322432 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" podStartSLOduration=1.322414485 podStartE2EDuration="1.322414485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:05:27.320454005 +0000 UTC m=+1032.774259964" watchObservedRunningTime="2026-09-11 19:05:27.322414485 +0000 UTC m=+1032.776220420" Sep 11 19:05:27.341164 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:27.341112 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" podStartSLOduration=1.341095859 podStartE2EDuration="1.341095859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:05:27.337403809 +0000 UTC m=+1032.791209764" watchObservedRunningTime="2026-09-11 19:05:27.341095859 +0000 UTC m=+1032.794901816" Sep 11 19:05:27.359651 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:27.359606 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" podStartSLOduration=1.8932653 podStartE2EDuration="2.359589259s" totalImagesPullingTime="466.323959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:05:27.357195289 +0000 UTC m=+1032.811001244" watchObservedRunningTime="2026-09-11 19:05:27.359589259 +0000 UTC m=+1032.813395214" Sep 11 19:05:34.332143 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:34.332115 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:05:39.328758 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:39.328713 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" Sep 11 19:05:39.329174 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:39.328912 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" Sep 11 19:05:46.524806 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:46.524776 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:48.179207 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:48.179180 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:49.066289 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:49.066255 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:49.850632 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:49.850590 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:50.178377 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:50.178298 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:50.855956 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:50.855923 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:51.139785 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:51.139713 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:51.801980 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:51.801943 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:51.896364 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:51.896328 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:52.000319 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:52.000289 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:52.232583 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:52.232429 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:52.888447 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:52.888418 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:53.668425 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:53.668394 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:54.484030 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:54.483998 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:55.266375 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:55.266347 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:55.358627 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:55.358592 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:56.253250 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:56.253218 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:57.928658 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:57.928628 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:58.634572 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:58.634525 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:58.734793 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:58.734754 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:05:59.237790 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:05:59.237760 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:00.078168 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:00.078133 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:00.780001 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:00.779964 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:01.379387 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.379351 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb"] Sep 11 19:06:01.379665 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.379617 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" podUID="c48b656e-477e-40c4-85c5-7f1e7ea6f2e6" containerName="maas-api" containerID="cri-o://0f9f59581f05da4a82dfc6ef90c0e9b50082bddb40e5f93b69d5b9262b2be18d" gracePeriod=30 Sep 11 19:06:01.424707 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.424678 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:01.456922 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.456892 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k"] Sep 11 19:06:01.457149 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.457124 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" podUID="d1198a82-8895-4c64-9569-4dfc4cf33f28" containerName="payload-processing" containerID="cri-o://5913c54128a5571637365853f192ade1759f58257c3e3a975fd137ca78bb06bc" gracePeriod=30 Sep 11 19:06:01.467387 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.467354 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv"] Sep 11 19:06:01.467655 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.467630 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" podUID="d25897cb-64ad-4470-a8fe-b25d65436112" containerName="payload-pre-processing" containerID="cri-o://c9e5f3c696b0de2d114f7cbf58364e217963facc316fc274b8eddf53a558afee" gracePeriod=30 Sep 11 19:06:01.505184 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.505150 2556 generic.go:356] "Generic (PLEG): container finished" podID="c48b656e-477e-40c4-85c5-7f1e7ea6f2e6" containerID="0f9f59581f05da4a82dfc6ef90c0e9b50082bddb40e5f93b69d5b9262b2be18d" exitCode=0 Sep 11 19:06:01.505322 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.505216 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" event={"ID":"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6","Type":"ContainerDied","Data":"0f9f59581f05da4a82dfc6ef90c0e9b50082bddb40e5f93b69d5b9262b2be18d"} Sep 11 19:06:01.527247 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.527204 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:01.624760 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.624655 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:01.718094 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.717913 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:01.731301 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.731270 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:06:01.739183 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.739159 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" Sep 11 19:06:01.769249 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.768927 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7gx5\" (UniqueName: \"kubernetes.io/projected/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-kube-api-access-g7gx5\") pod \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " Sep 11 19:06:01.769249 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.768989 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-maas-api-tls\") pod \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " Sep 11 19:06:01.769249 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.769029 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-metrics-certs\") pod \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " Sep 11 19:06:01.770375 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.769575 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-openshift-service-ca\") pod \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\" (UID: \"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6\") " Sep 11 19:06:01.770375 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.769638 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1198a82-8895-4c64-9569-4dfc4cf33f28-plugins-config-volume\") pod \"d1198a82-8895-4c64-9569-4dfc4cf33f28\" (UID: \"d1198a82-8895-4c64-9569-4dfc4cf33f28\") " Sep 11 19:06:01.770375 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.769676 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv9fw\" (UniqueName: \"kubernetes.io/projected/d1198a82-8895-4c64-9569-4dfc4cf33f28-kube-api-access-pv9fw\") pod \"d1198a82-8895-4c64-9569-4dfc4cf33f28\" (UID: \"d1198a82-8895-4c64-9569-4dfc4cf33f28\") " Sep 11 19:06:01.771628 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.771519 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1198a82-8895-4c64-9569-4dfc4cf33f28-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1198a82-8895-4c64-9569-4dfc4cf33f28" (UID: "d1198a82-8895-4c64-9569-4dfc4cf33f28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:06:01.771628 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.771584 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c48b656e-477e-40c4-85c5-7f1e7ea6f2e6" (UID: "c48b656e-477e-40c4-85c5-7f1e7ea6f2e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:06:01.772790 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.772718 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-kube-api-access-g7gx5" (OuterVolumeSpecName: "kube-api-access-g7gx5") pod "c48b656e-477e-40c4-85c5-7f1e7ea6f2e6" (UID: "c48b656e-477e-40c4-85c5-7f1e7ea6f2e6"). InnerVolumeSpecName "kube-api-access-g7gx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:06:01.772891 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.772812 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1198a82-8895-4c64-9569-4dfc4cf33f28-kube-api-access-pv9fw" (OuterVolumeSpecName: "kube-api-access-pv9fw") pod "d1198a82-8895-4c64-9569-4dfc4cf33f28" (UID: "d1198a82-8895-4c64-9569-4dfc4cf33f28"). InnerVolumeSpecName "kube-api-access-pv9fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:06:01.775027 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.774978 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c48b656e-477e-40c4-85c5-7f1e7ea6f2e6" (UID: "c48b656e-477e-40c4-85c5-7f1e7ea6f2e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:06:01.775283 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.775256 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c48b656e-477e-40c4-85c5-7f1e7ea6f2e6" (UID: "c48b656e-477e-40c4-85c5-7f1e7ea6f2e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:06:01.789145 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.789122 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" Sep 11 19:06:01.871045 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.871012 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25897cb-64ad-4470-a8fe-b25d65436112-plugins-config-volume\") pod \"d25897cb-64ad-4470-a8fe-b25d65436112\" (UID: \"d25897cb-64ad-4470-a8fe-b25d65436112\") " Sep 11 19:06:01.871208 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.871158 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knlpn\" (UniqueName: \"kubernetes.io/projected/d25897cb-64ad-4470-a8fe-b25d65436112-kube-api-access-knlpn\") pod \"d25897cb-64ad-4470-a8fe-b25d65436112\" (UID: \"d25897cb-64ad-4470-a8fe-b25d65436112\") " Sep 11 19:06:01.871372 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.871356 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1198a82-8895-4c64-9569-4dfc4cf33f28-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:01.871422 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.871380 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv9fw\" (UniqueName: \"kubernetes.io/projected/d1198a82-8895-4c64-9569-4dfc4cf33f28-kube-api-access-pv9fw\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:01.871422 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.871394 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7gx5\" (UniqueName: \"kubernetes.io/projected/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-kube-api-access-g7gx5\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:01.871422 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.871408 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-maas-api-tls\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:01.871532 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.871422 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-metrics-certs\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:01.871532 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.871418 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25897cb-64ad-4470-a8fe-b25d65436112-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d25897cb-64ad-4470-a8fe-b25d65436112" (UID: "d25897cb-64ad-4470-a8fe-b25d65436112"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:06:01.871532 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.871434 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6-openshift-service-ca\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:01.873201 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.873173 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25897cb-64ad-4470-a8fe-b25d65436112-kube-api-access-knlpn" (OuterVolumeSpecName: "kube-api-access-knlpn") pod "d25897cb-64ad-4470-a8fe-b25d65436112" (UID: "d25897cb-64ad-4470-a8fe-b25d65436112"). InnerVolumeSpecName "kube-api-access-knlpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:06:01.937176 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.937084 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:01.972780 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.972749 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knlpn\" (UniqueName: \"kubernetes.io/projected/d25897cb-64ad-4470-a8fe-b25d65436112-kube-api-access-knlpn\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:01.972780 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:01.972773 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25897cb-64ad-4470-a8fe-b25d65436112-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:02.064735 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.064704 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:02.511913 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.511879 2556 generic.go:356] "Generic (PLEG): container finished" podID="d25897cb-64ad-4470-a8fe-b25d65436112" containerID="c9e5f3c696b0de2d114f7cbf58364e217963facc316fc274b8eddf53a558afee" exitCode=0 Sep 11 19:06:02.512082 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.511920 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" Sep 11 19:06:02.512082 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.511970 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" event={"ID":"d25897cb-64ad-4470-a8fe-b25d65436112","Type":"ContainerDied","Data":"c9e5f3c696b0de2d114f7cbf58364e217963facc316fc274b8eddf53a558afee"} Sep 11 19:06:02.512082 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.512025 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv" event={"ID":"d25897cb-64ad-4470-a8fe-b25d65436112","Type":"ContainerDied","Data":"5dca09e1fcfd35526fb355a71ecc5d9284d832ef10fbf5291d25042f52b090b7"} Sep 11 19:06:02.512082 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.512052 2556 scope.go:117] "RemoveContainer" containerID="c9e5f3c696b0de2d114f7cbf58364e217963facc316fc274b8eddf53a558afee" Sep 11 19:06:02.513593 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.513575 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" Sep 11 19:06:02.513714 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.513577 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb" event={"ID":"c48b656e-477e-40c4-85c5-7f1e7ea6f2e6","Type":"ContainerDied","Data":"c5d2f75b5101f1c8374c4974f58910144af7d4b553f6a3fef8db5bc2ab639c07"} Sep 11 19:06:02.515184 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.515159 2556 generic.go:356] "Generic (PLEG): container finished" podID="d1198a82-8895-4c64-9569-4dfc4cf33f28" containerID="5913c54128a5571637365853f192ade1759f58257c3e3a975fd137ca78bb06bc" exitCode=0 Sep 11 19:06:02.515310 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.515196 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" event={"ID":"d1198a82-8895-4c64-9569-4dfc4cf33f28","Type":"ContainerDied","Data":"5913c54128a5571637365853f192ade1759f58257c3e3a975fd137ca78bb06bc"} Sep 11 19:06:02.515310 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.515206 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" Sep 11 19:06:02.515310 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.515232 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k" event={"ID":"d1198a82-8895-4c64-9569-4dfc4cf33f28","Type":"ContainerDied","Data":"bb3999c04ce1ff152d735b255c6649c80c9e71cde80e1810223e09f1664db1aa"} Sep 11 19:06:02.528167 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.528134 2556 scope.go:117] "RemoveContainer" containerID="c9e5f3c696b0de2d114f7cbf58364e217963facc316fc274b8eddf53a558afee" Sep 11 19:06:02.528580 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:06:02.528450 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e5f3c696b0de2d114f7cbf58364e217963facc316fc274b8eddf53a558afee\": container with ID starting with c9e5f3c696b0de2d114f7cbf58364e217963facc316fc274b8eddf53a558afee not found: ID does not exist" containerID="c9e5f3c696b0de2d114f7cbf58364e217963facc316fc274b8eddf53a558afee" Sep 11 19:06:02.528580 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.528484 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e5f3c696b0de2d114f7cbf58364e217963facc316fc274b8eddf53a558afee"} err="failed to get container status \"c9e5f3c696b0de2d114f7cbf58364e217963facc316fc274b8eddf53a558afee\": rpc error: code = NotFound desc = could not find container \"c9e5f3c696b0de2d114f7cbf58364e217963facc316fc274b8eddf53a558afee\": container with ID starting with c9e5f3c696b0de2d114f7cbf58364e217963facc316fc274b8eddf53a558afee not found: ID does not exist" Sep 11 19:06:02.528580 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.528507 2556 scope.go:117] "RemoveContainer" containerID="0f9f59581f05da4a82dfc6ef90c0e9b50082bddb40e5f93b69d5b9262b2be18d" Sep 11 19:06:02.538463 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.538446 2556 scope.go:117] "RemoveContainer" containerID="5913c54128a5571637365853f192ade1759f58257c3e3a975fd137ca78bb06bc" Sep 11 19:06:02.538639 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.538617 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv"] Sep 11 19:06:02.542745 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.542723 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f60615-7bf45f6cf8-qr6xv"] Sep 11 19:06:02.547042 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.547014 2556 scope.go:117] "RemoveContainer" containerID="5913c54128a5571637365853f192ade1759f58257c3e3a975fd137ca78bb06bc" Sep 11 19:06:02.547342 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:06:02.547316 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5913c54128a5571637365853f192ade1759f58257c3e3a975fd137ca78bb06bc\": container with ID starting with 5913c54128a5571637365853f192ade1759f58257c3e3a975fd137ca78bb06bc not found: ID does not exist" containerID="5913c54128a5571637365853f192ade1759f58257c3e3a975fd137ca78bb06bc" Sep 11 19:06:02.547416 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.547353 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5913c54128a5571637365853f192ade1759f58257c3e3a975fd137ca78bb06bc"} err="failed to get container status \"5913c54128a5571637365853f192ade1759f58257c3e3a975fd137ca78bb06bc\": rpc error: code = NotFound desc = could not find container \"5913c54128a5571637365853f192ade1759f58257c3e3a975fd137ca78bb06bc\": container with ID starting with 5913c54128a5571637365853f192ade1759f58257c3e3a975fd137ca78bb06bc not found: ID does not exist" Sep 11 19:06:02.554294 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.554274 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb"] Sep 11 19:06:02.560347 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.560320 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f60615-6488dc48c9-bnlkb"] Sep 11 19:06:02.573131 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.573110 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k"] Sep 11 19:06:02.581401 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.581377 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f60615-57548b9865-lnq8k"] Sep 11 19:06:02.934338 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:02.934308 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:03.172384 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:03.172350 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c48b656e-477e-40c4-85c5-7f1e7ea6f2e6" path="/var/lib/kubelet/pods/c48b656e-477e-40c4-85c5-7f1e7ea6f2e6/volumes" Sep 11 19:06:03.172777 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:03.172762 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1198a82-8895-4c64-9569-4dfc4cf33f28" path="/var/lib/kubelet/pods/d1198a82-8895-4c64-9569-4dfc4cf33f28/volumes" Sep 11 19:06:03.173082 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:03.173070 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d25897cb-64ad-4470-a8fe-b25d65436112" path="/var/lib/kubelet/pods/d25897cb-64ad-4470-a8fe-b25d65436112/volumes" Sep 11 19:06:03.777211 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:03.777176 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:04.457178 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:04.457133 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:05.379310 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:05.379268 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:06.080869 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:06.080830 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:06.921037 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:06.921005 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:07.681443 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:07.681406 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:08.322178 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:08.322142 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:08.424478 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:08.424442 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:08.563816 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:08.563777 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:08.668012 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:08.667893 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:08.967351 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:08.967263 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:09.225684 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.225494 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh"] Sep 11 19:06:09.225850 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.225747 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" podUID="c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af" containerName="maas-api" containerID="cri-o://5ea14a4134184d5c011e8244e48f39b9c1dccf6cf0954da7548e6ef85ecbeaee" gracePeriod=30 Sep 11 19:06:09.259250 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.259220 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:09.301318 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.301272 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj"] Sep 11 19:06:09.302457 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.301772 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" podUID="d500be3c-74c8-4e0d-9092-b93f0069a40d" containerName="payload-processing" containerID="cri-o://d72463236024a05e0d811861bafcf0993a2c2ff1234aa9819de9fb56f99422fe" gracePeriod=30 Sep 11 19:06:09.319550 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.319512 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285"] Sep 11 19:06:09.319763 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.319741 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" podUID="9b401684-b363-474a-8834-29dd7a6357c8" containerName="payload-pre-processing" containerID="cri-o://b7dd9c66372c8f3bd8624550251eeb2c0d36974ed2db9ceb8b028f16c76a4f86" gracePeriod=30 Sep 11 19:06:09.326971 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.326663 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" podUID="c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.55:8443/health\": dial tcp 10.132.0.55:8443: connect: connection refused" Sep 11 19:06:09.327736 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.327708 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" podUID="d500be3c-74c8-4e0d-9092-b93f0069a40d" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.56:9004: connect: connection refused" Sep 11 19:06:09.327958 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.327854 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" podUID="9b401684-b363-474a-8834-29dd7a6357c8" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.57:9004: connect: connection refused" Sep 11 19:06:09.350342 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.348470 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:09.419319 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.419284 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:09.518973 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.518942 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:09.564443 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.564412 2556 generic.go:356] "Generic (PLEG): container finished" podID="c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af" containerID="5ea14a4134184d5c011e8244e48f39b9c1dccf6cf0954da7548e6ef85ecbeaee" exitCode=0 Sep 11 19:06:09.564669 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.564574 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" event={"ID":"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af","Type":"ContainerDied","Data":"5ea14a4134184d5c011e8244e48f39b9c1dccf6cf0954da7548e6ef85ecbeaee"} Sep 11 19:06:09.564669 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.564630 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" event={"ID":"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af","Type":"ContainerDied","Data":"fc5b9ca0cf8ff31e97d9047c8047327ae393b7d3d3d44525d90f9a9aeeb240ec"} Sep 11 19:06:09.564669 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.564655 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc5b9ca0cf8ff31e97d9047c8047327ae393b7d3d3d44525d90f9a9aeeb240ec" Sep 11 19:06:09.567272 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.566868 2556 generic.go:356] "Generic (PLEG): container finished" podID="9b401684-b363-474a-8834-29dd7a6357c8" containerID="b7dd9c66372c8f3bd8624550251eeb2c0d36974ed2db9ceb8b028f16c76a4f86" exitCode=0 Sep 11 19:06:09.567272 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.566943 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" event={"ID":"9b401684-b363-474a-8834-29dd7a6357c8","Type":"ContainerDied","Data":"b7dd9c66372c8f3bd8624550251eeb2c0d36974ed2db9ceb8b028f16c76a4f86"} Sep 11 19:06:09.569228 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.569199 2556 generic.go:356] "Generic (PLEG): container finished" podID="d500be3c-74c8-4e0d-9092-b93f0069a40d" containerID="d72463236024a05e0d811861bafcf0993a2c2ff1234aa9819de9fb56f99422fe" exitCode=0 Sep 11 19:06:09.569348 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.569251 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" event={"ID":"d500be3c-74c8-4e0d-9092-b93f0069a40d","Type":"ContainerDied","Data":"d72463236024a05e0d811861bafcf0993a2c2ff1234aa9819de9fb56f99422fe"} Sep 11 19:06:09.576676 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.576653 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:06:09.585956 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.585904 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:09.630818 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.630796 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" Sep 11 19:06:09.633377 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.633359 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" Sep 11 19:06:09.642938 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.642916 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-openshift-service-ca\") pod \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " Sep 11 19:06:09.643054 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.642951 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-metrics-certs\") pod \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " Sep 11 19:06:09.643054 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.642983 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpw2p\" (UniqueName: \"kubernetes.io/projected/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-kube-api-access-fpw2p\") pod \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " Sep 11 19:06:09.643169 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.643055 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-maas-api-tls\") pod \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\" (UID: \"c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af\") " Sep 11 19:06:09.643594 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.643343 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af" (UID: "c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:06:09.644052 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.644032 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-openshift-service-ca\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:09.644992 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.644943 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:09.645997 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.645959 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af" (UID: "c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:06:09.646319 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.646287 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af" (UID: "c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:06:09.646623 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.646606 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-kube-api-access-fpw2p" (OuterVolumeSpecName: "kube-api-access-fpw2p") pod "c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af" (UID: "c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af"). InnerVolumeSpecName "kube-api-access-fpw2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:06:09.744864 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.744833 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d500be3c-74c8-4e0d-9092-b93f0069a40d-plugins-config-volume\") pod \"d500be3c-74c8-4e0d-9092-b93f0069a40d\" (UID: \"d500be3c-74c8-4e0d-9092-b93f0069a40d\") " Sep 11 19:06:09.745034 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.744982 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b401684-b363-474a-8834-29dd7a6357c8-plugins-config-volume\") pod \"9b401684-b363-474a-8834-29dd7a6357c8\" (UID: \"9b401684-b363-474a-8834-29dd7a6357c8\") " Sep 11 19:06:09.745117 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.745035 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9ht2\" (UniqueName: \"kubernetes.io/projected/d500be3c-74c8-4e0d-9092-b93f0069a40d-kube-api-access-l9ht2\") pod \"d500be3c-74c8-4e0d-9092-b93f0069a40d\" (UID: \"d500be3c-74c8-4e0d-9092-b93f0069a40d\") " Sep 11 19:06:09.745117 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.745059 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hhj5\" (UniqueName: \"kubernetes.io/projected/9b401684-b363-474a-8834-29dd7a6357c8-kube-api-access-2hhj5\") pod \"9b401684-b363-474a-8834-29dd7a6357c8\" (UID: \"9b401684-b363-474a-8834-29dd7a6357c8\") " Sep 11 19:06:09.745363 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.745339 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b401684-b363-474a-8834-29dd7a6357c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b401684-b363-474a-8834-29dd7a6357c8" (UID: "9b401684-b363-474a-8834-29dd7a6357c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:06:09.745363 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.745343 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d500be3c-74c8-4e0d-9092-b93f0069a40d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d500be3c-74c8-4e0d-9092-b93f0069a40d" (UID: "d500be3c-74c8-4e0d-9092-b93f0069a40d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:06:09.745475 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.745398 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-metrics-certs\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:09.745475 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.745419 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpw2p\" (UniqueName: \"kubernetes.io/projected/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-kube-api-access-fpw2p\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:09.745475 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.745432 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af-maas-api-tls\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:09.747150 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.747122 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b401684-b363-474a-8834-29dd7a6357c8-kube-api-access-2hhj5" (OuterVolumeSpecName: "kube-api-access-2hhj5") pod "9b401684-b363-474a-8834-29dd7a6357c8" (UID: "9b401684-b363-474a-8834-29dd7a6357c8"). InnerVolumeSpecName "kube-api-access-2hhj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:06:09.747274 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.747195 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d500be3c-74c8-4e0d-9092-b93f0069a40d-kube-api-access-l9ht2" (OuterVolumeSpecName: "kube-api-access-l9ht2") pod "d500be3c-74c8-4e0d-9092-b93f0069a40d" (UID: "d500be3c-74c8-4e0d-9092-b93f0069a40d"). InnerVolumeSpecName "kube-api-access-l9ht2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:06:09.846917 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.846887 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9ht2\" (UniqueName: \"kubernetes.io/projected/d500be3c-74c8-4e0d-9092-b93f0069a40d-kube-api-access-l9ht2\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:09.846917 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.846917 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hhj5\" (UniqueName: \"kubernetes.io/projected/9b401684-b363-474a-8834-29dd7a6357c8-kube-api-access-2hhj5\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:09.847111 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.846930 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d500be3c-74c8-4e0d-9092-b93f0069a40d-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:09.847111 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.846943 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b401684-b363-474a-8834-29dd7a6357c8-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:09.867881 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:09.867856 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:10.448026 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.447980 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:10.555411 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.555361 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:10.575804 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.575779 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" Sep 11 19:06:10.575963 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.575805 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj" event={"ID":"d500be3c-74c8-4e0d-9092-b93f0069a40d","Type":"ContainerDied","Data":"b5a211cc48de114bfc3b18479911ea51686e3a320a4008c390dca6d732b0d80c"} Sep 11 19:06:10.575963 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.575857 2556 scope.go:117] "RemoveContainer" containerID="d72463236024a05e0d811861bafcf0993a2c2ff1234aa9819de9fb56f99422fe" Sep 11 19:06:10.577307 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.577288 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" Sep 11 19:06:10.577388 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.577305 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285" event={"ID":"9b401684-b363-474a-8834-29dd7a6357c8","Type":"ContainerDied","Data":"bf8bbdb8bbf72a30585044bd331ad571b9625c6741c81f03467b2293557ec947"} Sep 11 19:06:10.577388 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.577316 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh" Sep 11 19:06:10.586534 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.586511 2556 scope.go:117] "RemoveContainer" containerID="b7dd9c66372c8f3bd8624550251eeb2c0d36974ed2db9ceb8b028f16c76a4f86" Sep 11 19:06:10.612891 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.612869 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285"] Sep 11 19:06:10.619717 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.619696 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a60dde0-6596756674-zq285"] Sep 11 19:06:10.631528 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.631508 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh"] Sep 11 19:06:10.639054 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.639016 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9a60dde0-84b5fcd545-wwqxh"] Sep 11 19:06:10.639054 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.639032 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:10.647749 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.647725 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj"] Sep 11 19:06:10.655359 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.655335 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a60dde0-7f7d5f4dd5-rlpsj"] Sep 11 19:06:10.743942 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:10.743912 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:11.172450 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:11.172414 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b401684-b363-474a-8834-29dd7a6357c8" path="/var/lib/kubelet/pods/9b401684-b363-474a-8834-29dd7a6357c8/volumes" Sep 11 19:06:11.172846 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:11.172832 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af" path="/var/lib/kubelet/pods/c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af/volumes" Sep 11 19:06:11.173149 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:11.173137 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d500be3c-74c8-4e0d-9092-b93f0069a40d" path="/var/lib/kubelet/pods/d500be3c-74c8-4e0d-9092-b93f0069a40d/volumes" Sep 11 19:06:11.688328 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:11.688297 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:12.571022 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:12.570991 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:13.276171 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:13.276139 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:14.180097 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:14.180066 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:16.301285 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:16.301252 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:16.400226 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:16.400192 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:16.561993 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:16.561892 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:16.981952 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:16.981843 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:17.079255 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:17.079219 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:17.174665 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:17.174629 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:23.286186 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:23.286150 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:23.391473 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:23.391442 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:23.492630 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:23.492600 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:23.636829 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:23.636744 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:27.623958 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:27.623922 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:28.808271 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808234 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz"] Sep 11 19:06:28.808663 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808649 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c48b656e-477e-40c4-85c5-7f1e7ea6f2e6" containerName="maas-api" Sep 11 19:06:28.808712 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808665 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c48b656e-477e-40c4-85c5-7f1e7ea6f2e6" containerName="maas-api" Sep 11 19:06:28.808712 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808679 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d25897cb-64ad-4470-a8fe-b25d65436112" containerName="payload-pre-processing" Sep 11 19:06:28.808712 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808688 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d25897cb-64ad-4470-a8fe-b25d65436112" containerName="payload-pre-processing" Sep 11 19:06:28.808712 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808694 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b401684-b363-474a-8834-29dd7a6357c8" containerName="payload-pre-processing" Sep 11 19:06:28.808712 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808701 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b401684-b363-474a-8834-29dd7a6357c8" containerName="payload-pre-processing" Sep 11 19:06:28.808712 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808709 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af" containerName="maas-api" Sep 11 19:06:28.808886 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808715 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af" containerName="maas-api" Sep 11 19:06:28.808886 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808724 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d500be3c-74c8-4e0d-9092-b93f0069a40d" containerName="payload-processing" Sep 11 19:06:28.808886 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808729 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d500be3c-74c8-4e0d-9092-b93f0069a40d" containerName="payload-processing" Sep 11 19:06:28.808886 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808754 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1198a82-8895-4c64-9569-4dfc4cf33f28" containerName="payload-processing" Sep 11 19:06:28.808886 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808759 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1198a82-8895-4c64-9569-4dfc4cf33f28" containerName="payload-processing" Sep 11 19:06:28.808886 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808817 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d25897cb-64ad-4470-a8fe-b25d65436112" containerName="payload-pre-processing" Sep 11 19:06:28.808886 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808826 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9cdfc5e-bcbb-4f79-9fc3-6a58f388b0af" containerName="maas-api" Sep 11 19:06:28.808886 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808832 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c48b656e-477e-40c4-85c5-7f1e7ea6f2e6" containerName="maas-api" Sep 11 19:06:28.808886 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808842 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1198a82-8895-4c64-9569-4dfc4cf33f28" containerName="payload-processing" Sep 11 19:06:28.808886 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808848 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b401684-b363-474a-8834-29dd7a6357c8" containerName="payload-pre-processing" Sep 11 19:06:28.808886 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.808854 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d500be3c-74c8-4e0d-9092-b93f0069a40d" containerName="payload-processing" Sep 11 19:06:28.813838 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.813821 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz" Sep 11 19:06:28.816181 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.816157 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\"" Sep 11 19:06:28.819803 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.819778 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz"] Sep 11 19:06:28.924627 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:28.924592 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eba49d5-873b-4414-ab37-6bba4b2ceb9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz\" (UID: \"9eba49d5-873b-4414-ab37-6bba4b2ceb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz" Sep 11 19:06:29.025262 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:29.025218 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eba49d5-873b-4414-ab37-6bba4b2ceb9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz\" (UID: \"9eba49d5-873b-4414-ab37-6bba4b2ceb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz" Sep 11 19:06:29.025885 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:29.025855 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eba49d5-873b-4414-ab37-6bba4b2ceb9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz\" (UID: \"9eba49d5-873b-4414-ab37-6bba4b2ceb9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz" Sep 11 19:06:29.125960 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:29.125872 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz" Sep 11 19:06:29.142438 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:29.142407 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:29.400116 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:29.400089 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz"] Sep 11 19:06:29.402262 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:06:29.402236 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eba49d5_873b_4414_ab37_6bba4b2ceb9f.slice/crio-b60ff3a7a518aeb0c62915c3dd892124a8202dccef6826fb10c79c21035e7427 WatchSource:0}: Error finding container b60ff3a7a518aeb0c62915c3dd892124a8202dccef6826fb10c79c21035e7427: Status 404 returned error can't find the container with id b60ff3a7a518aeb0c62915c3dd892124a8202dccef6826fb10c79c21035e7427 Sep 11 19:06:29.682311 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:29.682275 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz" event={"ID":"9eba49d5-873b-4414-ab37-6bba4b2ceb9f","Type":"ContainerStarted","Data":"3128023bc5d6662026693158fc29a78b8cfde76b1dba2e464514dd47073bbec1"} Sep 11 19:06:29.682465 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:29.682335 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz" event={"ID":"9eba49d5-873b-4414-ab37-6bba4b2ceb9f","Type":"ContainerStarted","Data":"b60ff3a7a518aeb0c62915c3dd892124a8202dccef6826fb10c79c21035e7427"} Sep 11 19:06:29.700456 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:29.700405 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz" podStartSLOduration=1.700389515 podStartE2EDuration="1.700389515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:06:29.697706832 +0000 UTC m=+1095.151512786" watchObservedRunningTime="2026-09-11 19:06:29.700389515 +0000 UTC m=+1095.154195503" Sep 11 19:06:29.992356 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:29.992252 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:30.181933 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:30.181898 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:30.280859 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:30.280762 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:30.690120 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:30.690087 2556 generic.go:356] "Generic (PLEG): container finished" podID="9eba49d5-873b-4414-ab37-6bba4b2ceb9f" containerID="3128023bc5d6662026693158fc29a78b8cfde76b1dba2e464514dd47073bbec1" exitCode=0 Sep 11 19:06:30.690296 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:30.690169 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz" event={"ID":"9eba49d5-873b-4414-ab37-6bba4b2ceb9f","Type":"ContainerDied","Data":"3128023bc5d6662026693158fc29a78b8cfde76b1dba2e464514dd47073bbec1"} Sep 11 19:06:30.809963 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:30.809927 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:31.061034 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:31.060995 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:31.403015 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:31.402923 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:31.627424 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:31.627388 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:31.789567 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:31.789515 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:31.822787 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:31.822766 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz" Sep 11 19:06:31.958991 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:31.958911 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eba49d5-873b-4414-ab37-6bba4b2ceb9f-maas-api-service-ca\") pod \"9eba49d5-873b-4414-ab37-6bba4b2ceb9f\" (UID: \"9eba49d5-873b-4414-ab37-6bba4b2ceb9f\") " Sep 11 19:06:31.959274 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:31.959252 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eba49d5-873b-4414-ab37-6bba4b2ceb9f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9eba49d5-873b-4414-ab37-6bba4b2ceb9f" (UID: "9eba49d5-873b-4414-ab37-6bba4b2ceb9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:06:32.059769 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:32.059732 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eba49d5-873b-4414-ab37-6bba4b2ceb9f-maas-api-service-ca\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:32.709172 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:32.709132 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz" event={"ID":"9eba49d5-873b-4414-ab37-6bba4b2ceb9f","Type":"ContainerDied","Data":"b60ff3a7a518aeb0c62915c3dd892124a8202dccef6826fb10c79c21035e7427"} Sep 11 19:06:32.709523 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:32.709222 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b60ff3a7a518aeb0c62915c3dd892124a8202dccef6826fb10c79c21035e7427" Sep 11 19:06:32.709523 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:32.709146 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz" Sep 11 19:06:32.807425 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:32.807397 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:33.706156 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:33.706121 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:34.361051 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:34.361013 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:34.447191 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:34.447134 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:34.804735 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:34.804255 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:35.584433 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:35.584395 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:36.367918 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:36.367883 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:37.261275 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:37.261242 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:37.372315 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:37.372281 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:38.051551 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.051465 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc"] Sep 11 19:06:38.052416 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.052387 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eba49d5-873b-4414-ab37-6bba4b2ceb9f" containerName="revoke-keys" Sep 11 19:06:38.052416 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.052419 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eba49d5-873b-4414-ab37-6bba4b2ceb9f" containerName="revoke-keys" Sep 11 19:06:38.052607 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.052569 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eba49d5-873b-4414-ab37-6bba4b2ceb9f" containerName="revoke-keys" Sep 11 19:06:38.056284 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.056257 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:06:38.059258 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.059235 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-19647e-serving-cert\"" Sep 11 19:06:38.059367 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.059310 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 19:06:38.059499 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.059477 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6bftc\"" Sep 11 19:06:38.069184 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.069160 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc"] Sep 11 19:06:38.099165 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.099119 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:38.116547 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.116505 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc\" (UID: \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:06:38.116646 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.116585 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvn8\" (UniqueName: \"kubernetes.io/projected/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-kube-api-access-dlvn8\") pod \"maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc\" (UID: \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:06:38.116704 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.116641 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc\" (UID: \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:06:38.116704 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.116680 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc\" (UID: \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:06:38.199716 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.199684 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:38.217636 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.217616 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc\" (UID: \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:06:38.217759 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.217644 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc\" (UID: \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:06:38.217759 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.217736 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc\" (UID: \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:06:38.217843 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.217777 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvn8\" (UniqueName: \"kubernetes.io/projected/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-kube-api-access-dlvn8\") pod \"maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc\" (UID: \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:06:38.218412 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.218385 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc\" (UID: \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:06:38.220132 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.220104 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc\" (UID: \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:06:38.220234 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.220142 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc\" (UID: \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:06:38.226036 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.226013 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvn8\" (UniqueName: \"kubernetes.io/projected/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-kube-api-access-dlvn8\") pod \"maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc\" (UID: \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:06:38.266442 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.266416 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:38.368484 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.368407 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:06:38.641402 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.641374 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc"] Sep 11 19:06:38.643476 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:06:38.643447 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce37bbd2_eb10_4c58_91c5_fdf15fa56810.slice/crio-92271e37b1ef4879a500c708a4ebb71ec14ee7f6088a94cbaa914d74334135ff WatchSource:0}: Error finding container 92271e37b1ef4879a500c708a4ebb71ec14ee7f6088a94cbaa914d74334135ff: Status 404 returned error can't find the container with id 92271e37b1ef4879a500c708a4ebb71ec14ee7f6088a94cbaa914d74334135ff Sep 11 19:06:38.700942 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.700914 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:38.743478 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.743448 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" event={"ID":"ce37bbd2-eb10-4c58-91c5-fdf15fa56810","Type":"ContainerStarted","Data":"92271e37b1ef4879a500c708a4ebb71ec14ee7f6088a94cbaa914d74334135ff"} Sep 11 19:06:38.830997 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.830963 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq"] Sep 11 19:06:38.835671 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.835655 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" Sep 11 19:06:38.838374 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.838350 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-19647e\"" Sep 11 19:06:38.849386 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.849361 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:38.857559 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.857519 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq"] Sep 11 19:06:38.921979 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.921894 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp"] Sep 11 19:06:38.924921 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.924889 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5dca55-4746-454a-8746-2c54d3f1fa60-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq\" (UID: \"7e5dca55-4746-454a-8746-2c54d3f1fa60\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" Sep 11 19:06:38.925056 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.924975 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq5kl\" (UniqueName: \"kubernetes.io/projected/7e5dca55-4746-454a-8746-2c54d3f1fa60-kube-api-access-tq5kl\") pod \"payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq\" (UID: \"7e5dca55-4746-454a-8746-2c54d3f1fa60\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" Sep 11 19:06:38.925440 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.925421 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" Sep 11 19:06:38.928192 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.928171 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-19647e-dockercfg-wcpzk\"" Sep 11 19:06:38.945608 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:38.945577 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp"] Sep 11 19:06:39.026325 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.026291 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5dca55-4746-454a-8746-2c54d3f1fa60-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq\" (UID: \"7e5dca55-4746-454a-8746-2c54d3f1fa60\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" Sep 11 19:06:39.026495 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.026336 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d67ab2c-ee3a-448c-8374-16e1bc25e089-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp\" (UID: \"9d67ab2c-ee3a-448c-8374-16e1bc25e089\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" Sep 11 19:06:39.026495 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.026441 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdv6h\" (UniqueName: \"kubernetes.io/projected/9d67ab2c-ee3a-448c-8374-16e1bc25e089-kube-api-access-rdv6h\") pod \"payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp\" (UID: \"9d67ab2c-ee3a-448c-8374-16e1bc25e089\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" Sep 11 19:06:39.026601 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.026499 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq5kl\" (UniqueName: \"kubernetes.io/projected/7e5dca55-4746-454a-8746-2c54d3f1fa60-kube-api-access-tq5kl\") pod \"payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq\" (UID: \"7e5dca55-4746-454a-8746-2c54d3f1fa60\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" Sep 11 19:06:39.027004 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.026984 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5dca55-4746-454a-8746-2c54d3f1fa60-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq\" (UID: \"7e5dca55-4746-454a-8746-2c54d3f1fa60\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" Sep 11 19:06:39.035074 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.035054 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq5kl\" (UniqueName: \"kubernetes.io/projected/7e5dca55-4746-454a-8746-2c54d3f1fa60-kube-api-access-tq5kl\") pod \"payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq\" (UID: \"7e5dca55-4746-454a-8746-2c54d3f1fa60\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" Sep 11 19:06:39.126210 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.126153 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:39.127183 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.127158 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdv6h\" (UniqueName: \"kubernetes.io/projected/9d67ab2c-ee3a-448c-8374-16e1bc25e089-kube-api-access-rdv6h\") pod \"payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp\" (UID: \"9d67ab2c-ee3a-448c-8374-16e1bc25e089\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" Sep 11 19:06:39.127279 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.127269 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d67ab2c-ee3a-448c-8374-16e1bc25e089-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp\" (UID: \"9d67ab2c-ee3a-448c-8374-16e1bc25e089\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" Sep 11 19:06:39.127888 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.127871 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d67ab2c-ee3a-448c-8374-16e1bc25e089-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp\" (UID: \"9d67ab2c-ee3a-448c-8374-16e1bc25e089\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" Sep 11 19:06:39.135382 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.135359 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdv6h\" (UniqueName: \"kubernetes.io/projected/9d67ab2c-ee3a-448c-8374-16e1bc25e089-kube-api-access-rdv6h\") pod \"payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp\" (UID: \"9d67ab2c-ee3a-448c-8374-16e1bc25e089\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" Sep 11 19:06:39.146510 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.146485 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" Sep 11 19:06:39.164025 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.163994 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp"] Sep 11 19:06:39.164239 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.164214 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" podUID="697928b0-6a79-4a45-9d7a-a84ed8941237" containerName="payload-pre-processing" containerID="cri-o://be50a82f55fbe8b5600b410c66eb384335a409c419413768de0d8b0495b426f3" gracePeriod=30 Sep 11 19:06:39.207280 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.207233 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:39.235321 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.235294 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" Sep 11 19:06:39.314100 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.314065 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:39.450948 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.450925 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" Sep 11 19:06:39.457769 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.457706 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq"] Sep 11 19:06:39.460143 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:06:39.460115 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5dca55_4746_454a_8746_2c54d3f1fa60.slice/crio-f5ccbc6a5d2eae8953a4a3028683a348da9f933d0bbb7af7363d233fdc5b66e6 WatchSource:0}: Error finding container f5ccbc6a5d2eae8953a4a3028683a348da9f933d0bbb7af7363d233fdc5b66e6: Status 404 returned error can't find the container with id f5ccbc6a5d2eae8953a4a3028683a348da9f933d0bbb7af7363d233fdc5b66e6 Sep 11 19:06:39.531019 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.530993 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-br4g7\" (UniqueName: \"kubernetes.io/projected/697928b0-6a79-4a45-9d7a-a84ed8941237-kube-api-access-br4g7\") pod \"697928b0-6a79-4a45-9d7a-a84ed8941237\" (UID: \"697928b0-6a79-4a45-9d7a-a84ed8941237\") " Sep 11 19:06:39.531156 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.531110 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/697928b0-6a79-4a45-9d7a-a84ed8941237-plugins-config-volume\") pod \"697928b0-6a79-4a45-9d7a-a84ed8941237\" (UID: \"697928b0-6a79-4a45-9d7a-a84ed8941237\") " Sep 11 19:06:39.531696 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.531641 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/697928b0-6a79-4a45-9d7a-a84ed8941237-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "697928b0-6a79-4a45-9d7a-a84ed8941237" (UID: "697928b0-6a79-4a45-9d7a-a84ed8941237"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:06:39.533131 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.533109 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697928b0-6a79-4a45-9d7a-a84ed8941237-kube-api-access-br4g7" (OuterVolumeSpecName: "kube-api-access-br4g7") pod "697928b0-6a79-4a45-9d7a-a84ed8941237" (UID: "697928b0-6a79-4a45-9d7a-a84ed8941237"). InnerVolumeSpecName "kube-api-access-br4g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:06:39.546090 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.546053 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:39.562361 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.562332 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz"] Sep 11 19:06:39.573420 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.573394 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp"] Sep 11 19:06:39.577462 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:06:39.577436 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d67ab2c_ee3a_448c_8374_16e1bc25e089.slice/crio-503b01b8642e28b228df2981dd278b365cb39816f4d6118814d241d44409dd84 WatchSource:0}: Error finding container 503b01b8642e28b228df2981dd278b365cb39816f4d6118814d241d44409dd84: Status 404 returned error can't find the container with id 503b01b8642e28b228df2981dd278b365cb39816f4d6118814d241d44409dd84 Sep 11 19:06:39.582332 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.582308 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c6c610-7wnnz"] Sep 11 19:06:39.632779 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.632753 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br4g7\" (UniqueName: \"kubernetes.io/projected/697928b0-6a79-4a45-9d7a-a84ed8941237-kube-api-access-br4g7\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:39.632870 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.632783 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/697928b0-6a79-4a45-9d7a-a84ed8941237-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:06:39.650486 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.650402 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:39.751320 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.750997 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" event={"ID":"ce37bbd2-eb10-4c58-91c5-fdf15fa56810","Type":"ContainerStarted","Data":"acfee418d17ce3a069a7efc5288605a271f8cbdc46aff8e94ed3ea816a03d872"} Sep 11 19:06:39.751320 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.751218 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:06:39.753174 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.753059 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" event={"ID":"9d67ab2c-ee3a-448c-8374-16e1bc25e089","Type":"ContainerStarted","Data":"59f3a12f0e8c17fa26c4fc6533f8acefc12806d7a4fea3b89358a9a876fad517"} Sep 11 19:06:39.753174 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.753114 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" Sep 11 19:06:39.753174 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.753140 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" event={"ID":"9d67ab2c-ee3a-448c-8374-16e1bc25e089","Type":"ContainerStarted","Data":"503b01b8642e28b228df2981dd278b365cb39816f4d6118814d241d44409dd84"} Sep 11 19:06:39.754931 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.754901 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" event={"ID":"7e5dca55-4746-454a-8746-2c54d3f1fa60","Type":"ContainerStarted","Data":"e2813982ad58650924b56211125feb627738b474251cdf6af0d9c9d3aae90495"} Sep 11 19:06:39.755036 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.754943 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" event={"ID":"7e5dca55-4746-454a-8746-2c54d3f1fa60","Type":"ContainerStarted","Data":"f5ccbc6a5d2eae8953a4a3028683a348da9f933d0bbb7af7363d233fdc5b66e6"} Sep 11 19:06:39.755251 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.755206 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" Sep 11 19:06:39.756416 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.756391 2556 generic.go:356] "Generic (PLEG): container finished" podID="697928b0-6a79-4a45-9d7a-a84ed8941237" containerID="be50a82f55fbe8b5600b410c66eb384335a409c419413768de0d8b0495b426f3" exitCode=0 Sep 11 19:06:39.756514 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.756427 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" Sep 11 19:06:39.756514 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.756479 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" event={"ID":"697928b0-6a79-4a45-9d7a-a84ed8941237","Type":"ContainerDied","Data":"be50a82f55fbe8b5600b410c66eb384335a409c419413768de0d8b0495b426f3"} Sep 11 19:06:39.756630 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.756529 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp" event={"ID":"697928b0-6a79-4a45-9d7a-a84ed8941237","Type":"ContainerDied","Data":"f222c5721eed0833fa84a4be9906ba23758224add0f1889a6da85a38b5b87bde"} Sep 11 19:06:39.756630 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.756581 2556 scope.go:117] "RemoveContainer" containerID="be50a82f55fbe8b5600b410c66eb384335a409c419413768de0d8b0495b426f3" Sep 11 19:06:39.767207 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.767188 2556 scope.go:117] "RemoveContainer" containerID="be50a82f55fbe8b5600b410c66eb384335a409c419413768de0d8b0495b426f3" Sep 11 19:06:39.767479 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:06:39.767451 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be50a82f55fbe8b5600b410c66eb384335a409c419413768de0d8b0495b426f3\": container with ID starting with be50a82f55fbe8b5600b410c66eb384335a409c419413768de0d8b0495b426f3 not found: ID does not exist" containerID="be50a82f55fbe8b5600b410c66eb384335a409c419413768de0d8b0495b426f3" Sep 11 19:06:39.767585 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.767489 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be50a82f55fbe8b5600b410c66eb384335a409c419413768de0d8b0495b426f3"} err="failed to get container status \"be50a82f55fbe8b5600b410c66eb384335a409c419413768de0d8b0495b426f3\": rpc error: code = NotFound desc = could not find container \"be50a82f55fbe8b5600b410c66eb384335a409c419413768de0d8b0495b426f3\": container with ID starting with be50a82f55fbe8b5600b410c66eb384335a409c419413768de0d8b0495b426f3 not found: ID does not exist" Sep 11 19:06:39.775100 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.775046 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" podStartSLOduration=1.298391043 podStartE2EDuration="1.775027161s" totalImagesPullingTime="476.636118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:06:39.773422601 +0000 UTC m=+1105.227228557" watchObservedRunningTime="2026-09-11 19:06:39.775027161 +0000 UTC m=+1105.228833117" Sep 11 19:06:39.796149 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.796099 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" podStartSLOduration=1.796079882 podStartE2EDuration="1.796079882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:06:39.793615865 +0000 UTC m=+1105.247421845" watchObservedRunningTime="2026-09-11 19:06:39.796079882 +0000 UTC m=+1105.249885841" Sep 11 19:06:39.817203 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.817169 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp"] Sep 11 19:06:39.824860 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.824836 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c6c610-5fc577f878-9vstp"] Sep 11 19:06:39.843758 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.843698 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" podStartSLOduration=1.843680186 podStartE2EDuration="1.843680186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:06:39.84243635 +0000 UTC m=+1105.296242304" watchObservedRunningTime="2026-09-11 19:06:39.843680186 +0000 UTC m=+1105.297486139" Sep 11 19:06:39.866144 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:39.866113 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:40.426284 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:40.426247 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:41.172391 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:41.172351 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="697928b0-6a79-4a45-9d7a-a84ed8941237" path="/var/lib/kubelet/pods/697928b0-6a79-4a45-9d7a-a84ed8941237/volumes" Sep 11 19:06:41.172785 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:41.172770 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eba49d5-873b-4414-ab37-6bba4b2ceb9f" path="/var/lib/kubelet/pods/9eba49d5-873b-4414-ab37-6bba4b2ceb9f/volumes" Sep 11 19:06:41.260952 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:41.260896 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:42.183410 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:42.183372 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:43.077568 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:43.077510 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:43.838168 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:43.838133 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:45.768319 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:45.768289 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:06:46.237951 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:46.237914 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:46.335677 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:46.335643 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:46.483339 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:46.483291 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:47.108002 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:47.107967 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:48.053232 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:48.053184 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:48.816833 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:48.816799 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:49.595585 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:49.595548 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:50.468326 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:50.468294 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:50.765262 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:50.765179 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" Sep 11 19:06:50.765503 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:50.765486 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" Sep 11 19:06:51.331496 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:51.331456 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:52.157904 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:52.157861 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:53.178426 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:53.178339 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:53.974487 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:53.974451 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:54.147487 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:54.147455 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:54.471717 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:54.471686 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:54.803350 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:54.803317 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:54.898590 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:54.898554 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:55.157779 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:55.157698 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:56.005801 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:56.005773 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:57.197926 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:57.197893 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:57.635376 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:57.635322 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:57.724760 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:57.724722 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:58.404850 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:58.404815 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:06:59.308949 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:06:59.308898 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:00.201473 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:00.201440 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:00.950526 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:00.950493 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:00.986976 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:00.986937 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp"] Sep 11 19:07:00.987180 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:00.987158 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" podUID="74a89864-c5e3-42fe-9b71-4f0be63be7c9" containerName="payload-processing" containerID="cri-o://e5a557cf072c474293f35cef624e47e4bec31382a7376ebc96819e4d9c0c9a15" gracePeriod=30 Sep 11 19:07:01.002405 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.002373 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc"] Sep 11 19:07:01.002618 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.002595 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" podUID="da6213b5-0cb2-41f8-8e6a-66ed53bb3230" containerName="payload-pre-processing" containerID="cri-o://60bb260a0451302ab662d8847169e999d60e6f82f9f8c9c3a9c3732512850289" gracePeriod=30 Sep 11 19:07:01.052979 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.052937 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:01.152442 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.152406 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:01.229027 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.228992 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:01.276687 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.276663 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" Sep 11 19:07:01.288605 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.288583 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" Sep 11 19:07:01.349911 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.348289 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a89864-c5e3-42fe-9b71-4f0be63be7c9-plugins-config-volume\") pod \"74a89864-c5e3-42fe-9b71-4f0be63be7c9\" (UID: \"74a89864-c5e3-42fe-9b71-4f0be63be7c9\") " Sep 11 19:07:01.349911 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.348399 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99r7n\" (UniqueName: \"kubernetes.io/projected/74a89864-c5e3-42fe-9b71-4f0be63be7c9-kube-api-access-99r7n\") pod \"74a89864-c5e3-42fe-9b71-4f0be63be7c9\" (UID: \"74a89864-c5e3-42fe-9b71-4f0be63be7c9\") " Sep 11 19:07:01.349911 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.349802 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a89864-c5e3-42fe-9b71-4f0be63be7c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74a89864-c5e3-42fe-9b71-4f0be63be7c9" (UID: "74a89864-c5e3-42fe-9b71-4f0be63be7c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:07:01.351315 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.351276 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a89864-c5e3-42fe-9b71-4f0be63be7c9-kube-api-access-99r7n" (OuterVolumeSpecName: "kube-api-access-99r7n") pod "74a89864-c5e3-42fe-9b71-4f0be63be7c9" (UID: "74a89864-c5e3-42fe-9b71-4f0be63be7c9"). InnerVolumeSpecName "kube-api-access-99r7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:07:01.366806 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.366779 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:01.449308 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.449257 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6213b5-0cb2-41f8-8e6a-66ed53bb3230-plugins-config-volume\") pod \"da6213b5-0cb2-41f8-8e6a-66ed53bb3230\" (UID: \"da6213b5-0cb2-41f8-8e6a-66ed53bb3230\") " Sep 11 19:07:01.449526 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.449386 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff7xp\" (UniqueName: \"kubernetes.io/projected/da6213b5-0cb2-41f8-8e6a-66ed53bb3230-kube-api-access-ff7xp\") pod \"da6213b5-0cb2-41f8-8e6a-66ed53bb3230\" (UID: \"da6213b5-0cb2-41f8-8e6a-66ed53bb3230\") " Sep 11 19:07:01.449724 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.449651 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a89864-c5e3-42fe-9b71-4f0be63be7c9-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:07:01.449724 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.449676 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6213b5-0cb2-41f8-8e6a-66ed53bb3230-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da6213b5-0cb2-41f8-8e6a-66ed53bb3230" (UID: "da6213b5-0cb2-41f8-8e6a-66ed53bb3230"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:07:01.449724 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.449699 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99r7n\" (UniqueName: \"kubernetes.io/projected/74a89864-c5e3-42fe-9b71-4f0be63be7c9-kube-api-access-99r7n\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:07:01.453428 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.453403 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6213b5-0cb2-41f8-8e6a-66ed53bb3230-kube-api-access-ff7xp" (OuterVolumeSpecName: "kube-api-access-ff7xp") pod "da6213b5-0cb2-41f8-8e6a-66ed53bb3230" (UID: "da6213b5-0cb2-41f8-8e6a-66ed53bb3230"). InnerVolumeSpecName "kube-api-access-ff7xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:07:01.456446 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.456419 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn"] Sep 11 19:07:01.457079 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.457064 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6213b5-0cb2-41f8-8e6a-66ed53bb3230" containerName="payload-pre-processing" Sep 11 19:07:01.457143 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.457083 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6213b5-0cb2-41f8-8e6a-66ed53bb3230" containerName="payload-pre-processing" Sep 11 19:07:01.457143 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.457095 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a89864-c5e3-42fe-9b71-4f0be63be7c9" containerName="payload-processing" Sep 11 19:07:01.457143 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.457105 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a89864-c5e3-42fe-9b71-4f0be63be7c9" containerName="payload-processing" Sep 11 19:07:01.457143 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.457135 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="697928b0-6a79-4a45-9d7a-a84ed8941237" containerName="payload-pre-processing" Sep 11 19:07:01.457280 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.457144 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="697928b0-6a79-4a45-9d7a-a84ed8941237" containerName="payload-pre-processing" Sep 11 19:07:01.457280 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.457235 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a89864-c5e3-42fe-9b71-4f0be63be7c9" containerName="payload-processing" Sep 11 19:07:01.457280 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.457250 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6213b5-0cb2-41f8-8e6a-66ed53bb3230" containerName="payload-pre-processing" Sep 11 19:07:01.457280 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.457262 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="697928b0-6a79-4a45-9d7a-a84ed8941237" containerName="payload-pre-processing" Sep 11 19:07:01.461017 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.460965 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:01.463204 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.463185 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-37ce96-serving-cert\"" Sep 11 19:07:01.473981 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.473951 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn"] Sep 11 19:07:01.510200 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.510136 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:01.551037 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.551003 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70526a66-5d7f-4170-a649-2548ddbaac4a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:01.551243 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.551111 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22cvb\" (UniqueName: \"kubernetes.io/projected/70526a66-5d7f-4170-a649-2548ddbaac4a-kube-api-access-22cvb\") pod \"maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:01.551243 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.551133 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70526a66-5d7f-4170-a649-2548ddbaac4a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:01.551243 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.551154 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70526a66-5d7f-4170-a649-2548ddbaac4a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:01.551384 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.551272 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6213b5-0cb2-41f8-8e6a-66ed53bb3230-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:07:01.551384 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.551300 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff7xp\" (UniqueName: \"kubernetes.io/projected/da6213b5-0cb2-41f8-8e6a-66ed53bb3230-kube-api-access-ff7xp\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:07:01.611233 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.611198 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:01.652701 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.652664 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70526a66-5d7f-4170-a649-2548ddbaac4a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:01.652906 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.652749 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22cvb\" (UniqueName: \"kubernetes.io/projected/70526a66-5d7f-4170-a649-2548ddbaac4a-kube-api-access-22cvb\") pod \"maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:01.652906 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.652772 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70526a66-5d7f-4170-a649-2548ddbaac4a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:01.652906 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.652795 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70526a66-5d7f-4170-a649-2548ddbaac4a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:01.652906 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:07:01.652837 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-serving-cert: secret "maas-api-e2e-shared-b-w2-37ce96-serving-cert" not found Sep 11 19:07:01.653124 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:07:01.652920 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70526a66-5d7f-4170-a649-2548ddbaac4a-maas-api-tls podName:70526a66-5d7f-4170-a649-2548ddbaac4a nodeName:}" failed. No retries permitted until 2026-09-11 19:07:02.152897564 +0000 UTC m=+1127.606703497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70526a66-5d7f-4170-a649-2548ddbaac4a-maas-api-tls") pod "maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" (UID: "70526a66-5d7f-4170-a649-2548ddbaac4a") : secret "maas-api-e2e-shared-b-w2-37ce96-serving-cert" not found Sep 11 19:07:01.653499 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.653477 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70526a66-5d7f-4170-a649-2548ddbaac4a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:01.655270 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.655246 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70526a66-5d7f-4170-a649-2548ddbaac4a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:01.667715 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.667682 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22cvb\" (UniqueName: \"kubernetes.io/projected/70526a66-5d7f-4170-a649-2548ddbaac4a-kube-api-access-22cvb\") pod \"maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:01.715963 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.715875 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:01.817643 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.817605 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:01.880820 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.880784 2556 generic.go:356] "Generic (PLEG): container finished" podID="da6213b5-0cb2-41f8-8e6a-66ed53bb3230" containerID="60bb260a0451302ab662d8847169e999d60e6f82f9f8c9c3a9c3732512850289" exitCode=0 Sep 11 19:07:01.881013 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.880834 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" Sep 11 19:07:01.881013 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.880878 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" event={"ID":"da6213b5-0cb2-41f8-8e6a-66ed53bb3230","Type":"ContainerDied","Data":"60bb260a0451302ab662d8847169e999d60e6f82f9f8c9c3a9c3732512850289"} Sep 11 19:07:01.881013 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.880932 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc" event={"ID":"da6213b5-0cb2-41f8-8e6a-66ed53bb3230","Type":"ContainerDied","Data":"2da001c35197e639bf1aa5138735ff2c0488cbf3c97a67c54391140f4666d8a8"} Sep 11 19:07:01.881013 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.880962 2556 scope.go:117] "RemoveContainer" containerID="60bb260a0451302ab662d8847169e999d60e6f82f9f8c9c3a9c3732512850289" Sep 11 19:07:01.882219 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.882194 2556 generic.go:356] "Generic (PLEG): container finished" podID="74a89864-c5e3-42fe-9b71-4f0be63be7c9" containerID="e5a557cf072c474293f35cef624e47e4bec31382a7376ebc96819e4d9c0c9a15" exitCode=0 Sep 11 19:07:01.882326 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.882232 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" Sep 11 19:07:01.882326 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.882254 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" event={"ID":"74a89864-c5e3-42fe-9b71-4f0be63be7c9","Type":"ContainerDied","Data":"e5a557cf072c474293f35cef624e47e4bec31382a7376ebc96819e4d9c0c9a15"} Sep 11 19:07:01.882326 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.882289 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp" event={"ID":"74a89864-c5e3-42fe-9b71-4f0be63be7c9","Type":"ContainerDied","Data":"a86c7d81ebf335f503e262cf33d24fd815c57c0fb31addf515ced32fbc7628c8"} Sep 11 19:07:01.890825 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.890801 2556 scope.go:117] "RemoveContainer" containerID="60bb260a0451302ab662d8847169e999d60e6f82f9f8c9c3a9c3732512850289" Sep 11 19:07:01.891123 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:07:01.891099 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60bb260a0451302ab662d8847169e999d60e6f82f9f8c9c3a9c3732512850289\": container with ID starting with 60bb260a0451302ab662d8847169e999d60e6f82f9f8c9c3a9c3732512850289 not found: ID does not exist" containerID="60bb260a0451302ab662d8847169e999d60e6f82f9f8c9c3a9c3732512850289" Sep 11 19:07:01.891241 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.891132 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60bb260a0451302ab662d8847169e999d60e6f82f9f8c9c3a9c3732512850289"} err="failed to get container status \"60bb260a0451302ab662d8847169e999d60e6f82f9f8c9c3a9c3732512850289\": rpc error: code = NotFound desc = could not find container \"60bb260a0451302ab662d8847169e999d60e6f82f9f8c9c3a9c3732512850289\": container with ID starting with 60bb260a0451302ab662d8847169e999d60e6f82f9f8c9c3a9c3732512850289 not found: ID does not exist" Sep 11 19:07:01.891241 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.891154 2556 scope.go:117] "RemoveContainer" containerID="e5a557cf072c474293f35cef624e47e4bec31382a7376ebc96819e4d9c0c9a15" Sep 11 19:07:01.899367 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.899349 2556 scope.go:117] "RemoveContainer" containerID="e5a557cf072c474293f35cef624e47e4bec31382a7376ebc96819e4d9c0c9a15" Sep 11 19:07:01.899716 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:07:01.899695 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a557cf072c474293f35cef624e47e4bec31382a7376ebc96819e4d9c0c9a15\": container with ID starting with e5a557cf072c474293f35cef624e47e4bec31382a7376ebc96819e4d9c0c9a15 not found: ID does not exist" containerID="e5a557cf072c474293f35cef624e47e4bec31382a7376ebc96819e4d9c0c9a15" Sep 11 19:07:01.899779 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.899725 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a557cf072c474293f35cef624e47e4bec31382a7376ebc96819e4d9c0c9a15"} err="failed to get container status \"e5a557cf072c474293f35cef624e47e4bec31382a7376ebc96819e4d9c0c9a15\": rpc error: code = NotFound desc = could not find container \"e5a557cf072c474293f35cef624e47e4bec31382a7376ebc96819e4d9c0c9a15\": container with ID starting with e5a557cf072c474293f35cef624e47e4bec31382a7376ebc96819e4d9c0c9a15 not found: ID does not exist" Sep 11 19:07:01.905697 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.905673 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc"] Sep 11 19:07:01.908062 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.908041 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-093f6c-858444c8bb-8lbvc"] Sep 11 19:07:01.918927 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.918902 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp"] Sep 11 19:07:01.922950 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:01.922917 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-093f6c-95489bbfc-bzdnp"] Sep 11 19:07:02.157512 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.157475 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70526a66-5d7f-4170-a649-2548ddbaac4a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:02.160130 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.160104 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70526a66-5d7f-4170-a649-2548ddbaac4a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:02.205845 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.205807 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:02.301097 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.301067 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:02.378333 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.378298 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:02.395292 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.395266 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj"] Sep 11 19:07:02.399089 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.399065 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" Sep 11 19:07:02.401341 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.401321 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-37ce96-dockercfg-k6n5n\"" Sep 11 19:07:02.401716 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.401697 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-37ce96\"" Sep 11 19:07:02.417187 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.417018 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj"] Sep 11 19:07:02.561905 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.561806 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37e489f-3485-47cb-9c43-67f0efdfb8e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj\" (UID: \"a37e489f-3485-47cb-9c43-67f0efdfb8e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" Sep 11 19:07:02.561905 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.561877 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtm4s\" (UniqueName: \"kubernetes.io/projected/a37e489f-3485-47cb-9c43-67f0efdfb8e5-kube-api-access-xtm4s\") pod \"payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj\" (UID: \"a37e489f-3485-47cb-9c43-67f0efdfb8e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" Sep 11 19:07:02.663048 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.663004 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtm4s\" (UniqueName: \"kubernetes.io/projected/a37e489f-3485-47cb-9c43-67f0efdfb8e5-kube-api-access-xtm4s\") pod \"payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj\" (UID: \"a37e489f-3485-47cb-9c43-67f0efdfb8e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" Sep 11 19:07:02.663233 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.663148 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37e489f-3485-47cb-9c43-67f0efdfb8e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj\" (UID: \"a37e489f-3485-47cb-9c43-67f0efdfb8e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" Sep 11 19:07:02.664207 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.664109 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37e489f-3485-47cb-9c43-67f0efdfb8e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj\" (UID: \"a37e489f-3485-47cb-9c43-67f0efdfb8e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" Sep 11 19:07:02.665728 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.665698 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn"] Sep 11 19:07:02.668318 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:07:02.668291 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70526a66_5d7f_4170_a649_2548ddbaac4a.slice/crio-1decc335a1e9cdcc9033ca1001060cdfcd8fd4a1a0b905db7382044d8da645c4 WatchSource:0}: Error finding container 1decc335a1e9cdcc9033ca1001060cdfcd8fd4a1a0b905db7382044d8da645c4: Status 404 returned error can't find the container with id 1decc335a1e9cdcc9033ca1001060cdfcd8fd4a1a0b905db7382044d8da645c4 Sep 11 19:07:02.672652 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.672621 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtm4s\" (UniqueName: \"kubernetes.io/projected/a37e489f-3485-47cb-9c43-67f0efdfb8e5-kube-api-access-xtm4s\") pod \"payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj\" (UID: \"a37e489f-3485-47cb-9c43-67f0efdfb8e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" Sep 11 19:07:02.718375 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.718329 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" Sep 11 19:07:02.830354 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.830260 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:02.890513 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.890476 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" event={"ID":"70526a66-5d7f-4170-a649-2548ddbaac4a","Type":"ContainerStarted","Data":"1decc335a1e9cdcc9033ca1001060cdfcd8fd4a1a0b905db7382044d8da645c4"} Sep 11 19:07:02.999851 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:02.999818 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj"] Sep 11 19:07:03.001514 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:07:03.001485 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37e489f_3485_47cb_9c43_67f0efdfb8e5.slice/crio-52f2a96650f48bd318d3063c467d9f14b2946e23082fd004a667a40e902c950b WatchSource:0}: Error finding container 52f2a96650f48bd318d3063c467d9f14b2946e23082fd004a667a40e902c950b: Status 404 returned error can't find the container with id 52f2a96650f48bd318d3063c467d9f14b2946e23082fd004a667a40e902c950b Sep 11 19:07:03.172556 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:03.172518 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74a89864-c5e3-42fe-9b71-4f0be63be7c9" path="/var/lib/kubelet/pods/74a89864-c5e3-42fe-9b71-4f0be63be7c9/volumes" Sep 11 19:07:03.172998 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:03.172978 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da6213b5-0cb2-41f8-8e6a-66ed53bb3230" path="/var/lib/kubelet/pods/da6213b5-0cb2-41f8-8e6a-66ed53bb3230/volumes" Sep 11 19:07:03.749126 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:03.749080 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:03.898400 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:03.898365 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" event={"ID":"70526a66-5d7f-4170-a649-2548ddbaac4a","Type":"ContainerStarted","Data":"eeeb838b01b991459f78a5c943b7baedcdbb9dc6059ac73858ffb985797827c0"} Sep 11 19:07:03.898611 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:03.898430 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:03.900348 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:03.900321 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" event={"ID":"a37e489f-3485-47cb-9c43-67f0efdfb8e5","Type":"ContainerStarted","Data":"39ccf22d214c7e29c81470369358cab71bd5ce59a3a7784aed27aefc229f9831"} Sep 11 19:07:03.900471 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:03.900366 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" event={"ID":"a37e489f-3485-47cb-9c43-67f0efdfb8e5","Type":"ContainerStarted","Data":"52f2a96650f48bd318d3063c467d9f14b2946e23082fd004a667a40e902c950b"} Sep 11 19:07:03.900471 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:03.900409 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" Sep 11 19:07:03.918581 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:03.918510 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" podStartSLOduration=2.574133674 podStartE2EDuration="2.918494611s" totalImagesPullingTime="344.360937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:07:03.917744626 +0000 UTC m=+1129.371550600" watchObservedRunningTime="2026-09-11 19:07:03.918494611 +0000 UTC m=+1129.372300567" Sep 11 19:07:03.936904 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:03.936856 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" podStartSLOduration=1.9368419239999999 podStartE2EDuration="1.936841924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:07:03.936142912 +0000 UTC m=+1129.389948884" watchObservedRunningTime="2026-09-11 19:07:03.936841924 +0000 UTC m=+1129.390647879" Sep 11 19:07:04.749803 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:04.749760 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:05.326748 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:05.326714 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:05.701128 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:05.701039 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:06.867131 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:06.867095 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:08.143922 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:08.143883 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:08.222451 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:08.222417 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:08.412807 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:08.412726 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:08.947314 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:08.947279 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:09.292756 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:09.292708 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:09.912387 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:09.912359 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:07:10.524688 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:10.524653 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:12.088794 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:12.088755 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:12.629089 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:12.629052 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:12.722142 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:12.722106 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:13.012845 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:13.012760 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:14.212313 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:14.212278 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:14.908711 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:14.908672 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" Sep 11 19:07:15.216073 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:15.216039 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:16.262954 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:16.262917 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:17.045963 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.045930 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd"] Sep 11 19:07:17.051356 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.051334 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:17.053979 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.053957 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-8280eb-serving-cert\"" Sep 11 19:07:17.065852 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.065827 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd"] Sep 11 19:07:17.093163 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.093137 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:17.100902 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.100877 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqcbq\" (UniqueName: \"kubernetes.io/projected/668f5591-b871-4427-ac57-f52e2f7e8aa6-kube-api-access-zqcbq\") pod \"maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:17.101019 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.100939 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668f5591-b871-4427-ac57-f52e2f7e8aa6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:17.101019 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.101006 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668f5591-b871-4427-ac57-f52e2f7e8aa6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:17.101125 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.101048 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668f5591-b871-4427-ac57-f52e2f7e8aa6-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:17.193615 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.193581 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:17.202551 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.202520 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqcbq\" (UniqueName: \"kubernetes.io/projected/668f5591-b871-4427-ac57-f52e2f7e8aa6-kube-api-access-zqcbq\") pod \"maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:17.202690 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.202579 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668f5591-b871-4427-ac57-f52e2f7e8aa6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:17.202690 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.202602 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668f5591-b871-4427-ac57-f52e2f7e8aa6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:17.202690 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.202630 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668f5591-b871-4427-ac57-f52e2f7e8aa6-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:17.203212 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:07:17.203179 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-serving-cert: secret "maas-api-e2e-shared-a-w3-8280eb-serving-cert" not found Sep 11 19:07:17.203295 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.203228 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668f5591-b871-4427-ac57-f52e2f7e8aa6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:17.203295 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:07:17.203273 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/668f5591-b871-4427-ac57-f52e2f7e8aa6-maas-api-tls podName:668f5591-b871-4427-ac57-f52e2f7e8aa6 nodeName:}" failed. No retries permitted until 2026-09-11 19:07:17.703253085 +0000 UTC m=+1143.157059021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/668f5591-b871-4427-ac57-f52e2f7e8aa6-maas-api-tls") pod "maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" (UID: "668f5591-b871-4427-ac57-f52e2f7e8aa6") : secret "maas-api-e2e-shared-a-w3-8280eb-serving-cert" not found Sep 11 19:07:17.205119 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.205098 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668f5591-b871-4427-ac57-f52e2f7e8aa6-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:17.211626 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.211603 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqcbq\" (UniqueName: \"kubernetes.io/projected/668f5591-b871-4427-ac57-f52e2f7e8aa6-kube-api-access-zqcbq\") pod \"maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:17.296375 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.296281 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:17.394028 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.393994 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:17.493806 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.493767 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:17.637984 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.637898 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:17.706733 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.706698 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668f5591-b871-4427-ac57-f52e2f7e8aa6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:17.709018 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.708995 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668f5591-b871-4427-ac57-f52e2f7e8aa6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:17.779895 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.779864 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:17.927727 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.927644 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp"] Sep 11 19:07:17.931502 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.931473 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" Sep 11 19:07:17.934197 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.934174 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-8280eb\"" Sep 11 19:07:17.934318 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.934174 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-8280eb-dockercfg-mskwn\"" Sep 11 19:07:17.942919 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.942895 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp"] Sep 11 19:07:17.964851 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:17.964809 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:18.010103 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:18.010067 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ea7ca2-b697-43a8-ba83-777729547927-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp\" (UID: \"a0ea7ca2-b697-43a8-ba83-777729547927\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" Sep 11 19:07:18.010274 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:18.010128 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxn9d\" (UniqueName: \"kubernetes.io/projected/a0ea7ca2-b697-43a8-ba83-777729547927-kube-api-access-kxn9d\") pod \"payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp\" (UID: \"a0ea7ca2-b697-43a8-ba83-777729547927\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" Sep 11 19:07:18.111121 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:18.111087 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxn9d\" (UniqueName: \"kubernetes.io/projected/a0ea7ca2-b697-43a8-ba83-777729547927-kube-api-access-kxn9d\") pod \"payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp\" (UID: \"a0ea7ca2-b697-43a8-ba83-777729547927\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" Sep 11 19:07:18.111303 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:18.111195 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ea7ca2-b697-43a8-ba83-777729547927-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp\" (UID: \"a0ea7ca2-b697-43a8-ba83-777729547927\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" Sep 11 19:07:18.111868 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:18.111848 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ea7ca2-b697-43a8-ba83-777729547927-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp\" (UID: \"a0ea7ca2-b697-43a8-ba83-777729547927\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" Sep 11 19:07:18.119169 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:18.119150 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxn9d\" (UniqueName: \"kubernetes.io/projected/a0ea7ca2-b697-43a8-ba83-777729547927-kube-api-access-kxn9d\") pod \"payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp\" (UID: \"a0ea7ca2-b697-43a8-ba83-777729547927\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" Sep 11 19:07:18.243349 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:18.243325 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" Sep 11 19:07:18.244870 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:18.244842 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd"] Sep 11 19:07:18.245886 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:07:18.245850 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668f5591_b871_4427_ac57_f52e2f7e8aa6.slice/crio-eba9eae41aac0b7cb904c50127b0776f5df1d31d1ad09851ae0ad64851674f58 WatchSource:0}: Error finding container eba9eae41aac0b7cb904c50127b0776f5df1d31d1ad09851ae0ad64851674f58: Status 404 returned error can't find the container with id eba9eae41aac0b7cb904c50127b0776f5df1d31d1ad09851ae0ad64851674f58 Sep 11 19:07:18.366955 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:18.366921 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:18.521763 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:18.521735 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp"] Sep 11 19:07:18.523732 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:07:18.523700 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ea7ca2_b697_43a8_ba83_777729547927.slice/crio-9de877f2476fad9ab73d57fce6d6b60f93296be2cfd8edeb678834b82626cc0b WatchSource:0}: Error finding container 9de877f2476fad9ab73d57fce6d6b60f93296be2cfd8edeb678834b82626cc0b: Status 404 returned error can't find the container with id 9de877f2476fad9ab73d57fce6d6b60f93296be2cfd8edeb678834b82626cc0b Sep 11 19:07:18.995015 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:18.994981 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" event={"ID":"668f5591-b871-4427-ac57-f52e2f7e8aa6","Type":"ContainerStarted","Data":"41f1fbc9b8b863ccf3954dd617037d01436e8b20d732df49fd92ccd4f8bc4c80"} Sep 11 19:07:18.995232 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:18.995084 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" event={"ID":"668f5591-b871-4427-ac57-f52e2f7e8aa6","Type":"ContainerStarted","Data":"eba9eae41aac0b7cb904c50127b0776f5df1d31d1ad09851ae0ad64851674f58"} Sep 11 19:07:18.995232 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:18.995116 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:18.996634 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:18.996604 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" event={"ID":"a0ea7ca2-b697-43a8-ba83-777729547927","Type":"ContainerStarted","Data":"7ab7a88d6f12887aeed35851adc1656351fec4ed7779aec215603d5f174b72e5"} Sep 11 19:07:18.996747 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:18.996655 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" event={"ID":"a0ea7ca2-b697-43a8-ba83-777729547927","Type":"ContainerStarted","Data":"9de877f2476fad9ab73d57fce6d6b60f93296be2cfd8edeb678834b82626cc0b"} Sep 11 19:07:18.996747 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:18.996685 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" Sep 11 19:07:19.014266 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:19.014213 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" podStartSLOduration=1.66830019 podStartE2EDuration="2.014195821s" totalImagesPullingTime="345.895631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:07:19.01267695 +0000 UTC m=+1144.466482928" watchObservedRunningTime="2026-09-11 19:07:19.014195821 +0000 UTC m=+1144.468001767" Sep 11 19:07:19.033573 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:19.033503 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" podStartSLOduration=2.033485983 podStartE2EDuration="2.033485983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:07:19.030889889 +0000 UTC m=+1144.484695845" watchObservedRunningTime="2026-09-11 19:07:19.033485983 +0000 UTC m=+1144.487291938" Sep 11 19:07:19.188260 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:19.188226 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:20.099256 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:20.099226 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:21.029298 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:21.029269 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:21.895527 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:21.895492 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:22.893409 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:22.893374 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:23.798929 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:23.798887 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:25.008219 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:25.008191 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:07:25.539015 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:25.538977 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:25.640822 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:25.640785 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:29.114732 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:29.114697 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:29.422439 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:29.422352 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:29.603179 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:29.603146 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:30.005392 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:30.005363 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" Sep 11 19:07:32.283006 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:32.282974 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:32.857888 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:32.857846 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:32.965691 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:32.965660 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:33.340349 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:33.340309 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:34.317356 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:34.317322 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:35.265261 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:35.265227 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:36.107042 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:36.107005 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:36.919733 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:36.919699 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:37.545181 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:37.545152 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:37.879001 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:37.878903 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:37.883148 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:37.883121 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq"] Sep 11 19:07:37.888089 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:37.888072 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:07:37.890605 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:37.890585 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7ad53a9d-serving-cert\"" Sep 11 19:07:37.899174 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:37.899118 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq"] Sep 11 19:07:37.977937 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:37.977904 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:38.007816 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.007793 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4846292a-38b1-4749-a23e-8f3a2860193a-metrics-certs\") pod \"maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq\" (UID: \"4846292a-38b1-4749-a23e-8f3a2860193a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:07:38.007964 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.007909 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4846292a-38b1-4749-a23e-8f3a2860193a-maas-api-tls\") pod \"maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq\" (UID: \"4846292a-38b1-4749-a23e-8f3a2860193a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:07:38.008033 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.007965 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4846292a-38b1-4749-a23e-8f3a2860193a-openshift-service-ca\") pod \"maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq\" (UID: \"4846292a-38b1-4749-a23e-8f3a2860193a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:07:38.008033 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.007992 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mccx\" (UniqueName: \"kubernetes.io/projected/4846292a-38b1-4749-a23e-8f3a2860193a-kube-api-access-2mccx\") pod \"maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq\" (UID: \"4846292a-38b1-4749-a23e-8f3a2860193a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:07:38.108769 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.108734 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4846292a-38b1-4749-a23e-8f3a2860193a-maas-api-tls\") pod \"maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq\" (UID: \"4846292a-38b1-4749-a23e-8f3a2860193a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:07:38.108936 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.108791 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4846292a-38b1-4749-a23e-8f3a2860193a-openshift-service-ca\") pod \"maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq\" (UID: \"4846292a-38b1-4749-a23e-8f3a2860193a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:07:38.108936 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.108820 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mccx\" (UniqueName: \"kubernetes.io/projected/4846292a-38b1-4749-a23e-8f3a2860193a-kube-api-access-2mccx\") pod \"maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq\" (UID: \"4846292a-38b1-4749-a23e-8f3a2860193a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:07:38.108936 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.108925 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4846292a-38b1-4749-a23e-8f3a2860193a-metrics-certs\") pod \"maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq\" (UID: \"4846292a-38b1-4749-a23e-8f3a2860193a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:07:38.109604 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.109570 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4846292a-38b1-4749-a23e-8f3a2860193a-openshift-service-ca\") pod \"maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq\" (UID: \"4846292a-38b1-4749-a23e-8f3a2860193a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:07:38.111216 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.111196 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4846292a-38b1-4749-a23e-8f3a2860193a-maas-api-tls\") pod \"maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq\" (UID: \"4846292a-38b1-4749-a23e-8f3a2860193a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:07:38.111315 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.111235 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4846292a-38b1-4749-a23e-8f3a2860193a-metrics-certs\") pod \"maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq\" (UID: \"4846292a-38b1-4749-a23e-8f3a2860193a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:07:38.121239 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.121210 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mccx\" (UniqueName: \"kubernetes.io/projected/4846292a-38b1-4749-a23e-8f3a2860193a-kube-api-access-2mccx\") pod \"maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq\" (UID: \"4846292a-38b1-4749-a23e-8f3a2860193a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:07:38.177784 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.177705 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:38.200816 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.200789 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:07:38.305399 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.305356 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:38.403781 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.403749 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:38.480275 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.480250 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq"] Sep 11 19:07:38.482177 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:07:38.482155 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4846292a_38b1_4749_a23e_8f3a2860193a.slice/crio-d8c8f4b5a17d169b02be901ccbd45ed4d67337123e23f31b05567f6dd4bf834e WatchSource:0}: Error finding container d8c8f4b5a17d169b02be901ccbd45ed4d67337123e23f31b05567f6dd4bf834e: Status 404 returned error can't find the container with id d8c8f4b5a17d169b02be901ccbd45ed4d67337123e23f31b05567f6dd4bf834e Sep 11 19:07:38.676682 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.676649 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:38.778328 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:38.778249 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:39.113248 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:39.113201 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" event={"ID":"4846292a-38b1-4749-a23e-8f3a2860193a","Type":"ContainerStarted","Data":"1bfb84910201c1242fc39a007ba5475d1aced344b5a2d7c1d8e56da441f95769"} Sep 11 19:07:39.113696 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:39.113391 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:07:39.113696 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:39.113417 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" event={"ID":"4846292a-38b1-4749-a23e-8f3a2860193a","Type":"ContainerStarted","Data":"d8c8f4b5a17d169b02be901ccbd45ed4d67337123e23f31b05567f6dd4bf834e"} Sep 11 19:07:39.133363 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:39.133318 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" podStartSLOduration=1.6798327830000002 podStartE2EDuration="2.133302965s" totalImagesPullingTime="453.470182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:07:39.13219686 +0000 UTC m=+1164.586002839" watchObservedRunningTime="2026-09-11 19:07:39.133302965 +0000 UTC m=+1164.587108919" Sep 11 19:07:39.212685 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:39.212643 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:40.208042 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:40.208008 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:40.763567 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:40.763507 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:41.404866 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:41.404833 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:41.680968 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:41.680883 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:41.800145 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:41.800111 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:42.512783 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:42.512749 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:43.450295 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:43.450258 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:44.488874 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:44.488839 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:45.124231 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:45.124203 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:07:45.282645 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:45.282605 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:46.058555 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:46.058509 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:47.099095 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:47.099064 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:47.882999 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:47.882965 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:48.851063 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:48.851027 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:48.950409 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:48.950378 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:49.152265 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.152188 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:49.562804 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.562769 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:49.705388 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.705358 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:49.805800 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.805766 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh"] Sep 11 19:07:49.809490 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.809473 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" Sep 11 19:07:49.811748 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.811724 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-3b7391\"" Sep 11 19:07:49.811748 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.811747 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-3b7391-dockercfg-997wx\"" Sep 11 19:07:49.822426 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.822401 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh"] Sep 11 19:07:49.832877 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.832854 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936bb385-4848-4cf3-b33c-0a4366114713-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh\" (UID: \"936bb385-4848-4cf3-b33c-0a4366114713\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" Sep 11 19:07:49.832969 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.832912 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2qw8\" (UniqueName: \"kubernetes.io/projected/936bb385-4848-4cf3-b33c-0a4366114713-kube-api-access-q2qw8\") pod \"payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh\" (UID: \"936bb385-4848-4cf3-b33c-0a4366114713\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" Sep 11 19:07:49.862555 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.862516 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z"] Sep 11 19:07:49.866294 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.866273 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" Sep 11 19:07:49.878365 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.878334 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z"] Sep 11 19:07:49.933495 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.933462 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2qw8\" (UniqueName: \"kubernetes.io/projected/936bb385-4848-4cf3-b33c-0a4366114713-kube-api-access-q2qw8\") pod \"payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh\" (UID: \"936bb385-4848-4cf3-b33c-0a4366114713\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" Sep 11 19:07:49.933686 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.933507 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7xm2\" (UniqueName: \"kubernetes.io/projected/7097a4f9-7ad3-4910-b2ff-f206f17fa531-kube-api-access-l7xm2\") pod \"payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z\" (UID: \"7097a4f9-7ad3-4910-b2ff-f206f17fa531\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" Sep 11 19:07:49.933686 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.933530 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7097a4f9-7ad3-4910-b2ff-f206f17fa531-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z\" (UID: \"7097a4f9-7ad3-4910-b2ff-f206f17fa531\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" Sep 11 19:07:49.933686 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.933588 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936bb385-4848-4cf3-b33c-0a4366114713-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh\" (UID: \"936bb385-4848-4cf3-b33c-0a4366114713\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" Sep 11 19:07:49.934182 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.934163 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936bb385-4848-4cf3-b33c-0a4366114713-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh\" (UID: \"936bb385-4848-4cf3-b33c-0a4366114713\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" Sep 11 19:07:49.941673 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:49.941646 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2qw8\" (UniqueName: \"kubernetes.io/projected/936bb385-4848-4cf3-b33c-0a4366114713-kube-api-access-q2qw8\") pod \"payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh\" (UID: \"936bb385-4848-4cf3-b33c-0a4366114713\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" Sep 11 19:07:50.034050 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:50.034021 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7xm2\" (UniqueName: \"kubernetes.io/projected/7097a4f9-7ad3-4910-b2ff-f206f17fa531-kube-api-access-l7xm2\") pod \"payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z\" (UID: \"7097a4f9-7ad3-4910-b2ff-f206f17fa531\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" Sep 11 19:07:50.034181 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:50.034056 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7097a4f9-7ad3-4910-b2ff-f206f17fa531-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z\" (UID: \"7097a4f9-7ad3-4910-b2ff-f206f17fa531\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" Sep 11 19:07:50.034786 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:50.034762 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7097a4f9-7ad3-4910-b2ff-f206f17fa531-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z\" (UID: \"7097a4f9-7ad3-4910-b2ff-f206f17fa531\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" Sep 11 19:07:50.043234 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:50.043211 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7xm2\" (UniqueName: \"kubernetes.io/projected/7097a4f9-7ad3-4910-b2ff-f206f17fa531-kube-api-access-l7xm2\") pod \"payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z\" (UID: \"7097a4f9-7ad3-4910-b2ff-f206f17fa531\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" Sep 11 19:07:50.105289 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:50.105219 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:50.120045 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:50.120020 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" Sep 11 19:07:50.178413 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:50.178071 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" Sep 11 19:07:50.398917 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:50.398896 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh"] Sep 11 19:07:50.400999 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:07:50.400975 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936bb385_4848_4cf3_b33c_0a4366114713.slice/crio-4a63d259d89319e8e12f95b04b2dfefe4d87cf045fb4a6adc27b5f89a1c87bc0 WatchSource:0}: Error finding container 4a63d259d89319e8e12f95b04b2dfefe4d87cf045fb4a6adc27b5f89a1c87bc0: Status 404 returned error can't find the container with id 4a63d259d89319e8e12f95b04b2dfefe4d87cf045fb4a6adc27b5f89a1c87bc0 Sep 11 19:07:50.484416 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:50.484390 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z"] Sep 11 19:07:50.486939 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:07:50.486909 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7097a4f9_7ad3_4910_b2ff_f206f17fa531.slice/crio-0b6de8a2b4f05eca4ae8cf219b5528f47964b19b995ac4a13bda871e1bef9615 WatchSource:0}: Error finding container 0b6de8a2b4f05eca4ae8cf219b5528f47964b19b995ac4a13bda871e1bef9615: Status 404 returned error can't find the container with id 0b6de8a2b4f05eca4ae8cf219b5528f47964b19b995ac4a13bda871e1bef9615 Sep 11 19:07:51.030625 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:51.030589 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:51.190619 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:51.190586 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" event={"ID":"7097a4f9-7ad3-4910-b2ff-f206f17fa531","Type":"ContainerStarted","Data":"d73652dc6c63d196cef2c2f3d1c4d4a9fb5ab160530d97d05a6e24a161e57016"} Sep 11 19:07:51.190919 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:51.190894 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" Sep 11 19:07:51.190919 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:51.190930 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" event={"ID":"7097a4f9-7ad3-4910-b2ff-f206f17fa531","Type":"ContainerStarted","Data":"0b6de8a2b4f05eca4ae8cf219b5528f47964b19b995ac4a13bda871e1bef9615"} Sep 11 19:07:51.192356 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:51.192331 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" event={"ID":"936bb385-4848-4cf3-b33c-0a4366114713","Type":"ContainerStarted","Data":"168f68e8689a64e2a6a5d95547258506c614978cdc2a88f97b2fa7476e0765aa"} Sep 11 19:07:51.192471 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:51.192377 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" event={"ID":"936bb385-4848-4cf3-b33c-0a4366114713","Type":"ContainerStarted","Data":"4a63d259d89319e8e12f95b04b2dfefe4d87cf045fb4a6adc27b5f89a1c87bc0"} Sep 11 19:07:51.192471 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:51.192416 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" Sep 11 19:07:51.212443 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:51.212397 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" podStartSLOduration=2.212381659 podStartE2EDuration="2.212381659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:07:51.20996835 +0000 UTC m=+1176.663774312" watchObservedRunningTime="2026-09-11 19:07:51.212381659 +0000 UTC m=+1176.666187618" Sep 11 19:07:51.228247 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:51.228189 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" podStartSLOduration=2.228168908 podStartE2EDuration="2.228168908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:07:51.226124914 +0000 UTC m=+1176.679930885" watchObservedRunningTime="2026-09-11 19:07:51.228168908 +0000 UTC m=+1176.681974865" Sep 11 19:07:51.948359 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:51.948321 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:52.635354 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:52.635322 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:52.838400 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:52.838360 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:53.778834 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:53.778794 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:54.907390 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:54.907357 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:55.902252 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:55.902200 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:56.722742 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:56.722706 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:07:59.297744 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:07:59.297706 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:02.200886 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:02.200859 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" Sep 11 19:08:02.201274 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:02.200919 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" Sep 11 19:08:06.794451 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:06.794417 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:10.173646 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:10.173614 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:11.617730 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:11.617694 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:11.722963 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:11.722913 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:12.410337 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:12.410305 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:13.385601 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:13.385527 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:14.280650 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:14.280618 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:14.759475 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:14.759440 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:14.964247 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:14.964219 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:15.086859 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:15.086813 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 19:08:15.088915 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:15.088886 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:15.089059 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:15.088989 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 19:08:15.193090 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:15.193059 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:15.196962 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:15.196942 2556 scope.go:117] "RemoveContainer" containerID="cd6cbdfc4fa25fbac9944d4a94bdb0eeba226b1c29c453b94ad6df99384e07c9" Sep 11 19:08:15.635003 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:15.634974 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:16.281514 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:16.281480 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:17.189322 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:17.189289 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:18.008397 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:18.008357 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:20.113613 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:20.113574 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:39.041730 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:39.041696 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:41.970928 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:41.970883 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:42.070489 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:42.070453 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:57.016747 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:57.016710 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:08:59.461432 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:08:59.461397 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:08.841030 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:08.840954 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:08.940584 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:08.940551 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:15.216815 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:15.216787 2556 scope.go:117] "RemoveContainer" containerID="de41741bb5304ddbf6cdc5e9887d10bb52d2f5975c167c5a537c18fad8c7366e" Sep 11 19:09:26.445557 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:26.445507 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:27.495749 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:27.495705 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:42.300675 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:42.300640 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:43.175753 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:43.175717 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:43.538288 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:43.538254 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:44.636103 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:44.636060 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:45.482708 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:45.482655 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:46.328307 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:46.328270 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:47.332820 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:47.332785 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:48.206979 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:48.206945 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:48.287182 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:48.287148 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:48.749082 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:48.749045 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:49.291578 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:49.291524 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:50.154009 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:50.153959 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:50.989414 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:50.989376 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:59.207709 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.207672 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:59.259922 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.259889 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f"] Sep 11 19:09:59.260386 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.260355 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" podUID="a8d574dd-4362-4e57-8dc0-b483d339f838" containerName="payload-processing" containerID="cri-o://47e5d81353c8b3191583fdc01420acb7ace059941d0345194c870da81d1704ae" gracePeriod=30 Sep 11 19:09:59.277608 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.277577 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl"] Sep 11 19:09:59.277849 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.277800 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" podUID="bc6eaa7b-901d-4bf0-86ff-672169131846" containerName="payload-pre-processing" containerID="cri-o://f667eaf4bddb401042ed573021fb4e4285fbf6c72251a10299ec39606deb843d" gracePeriod=30 Sep 11 19:09:59.313844 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.313809 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:59.415990 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.415943 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:59.518301 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.518260 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:59.580771 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.580747 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" Sep 11 19:09:59.583942 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.583922 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" Sep 11 19:09:59.612753 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.612721 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:59.651139 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.651113 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f97nl\" (UniqueName: \"kubernetes.io/projected/a8d574dd-4362-4e57-8dc0-b483d339f838-kube-api-access-f97nl\") pod \"a8d574dd-4362-4e57-8dc0-b483d339f838\" (UID: \"a8d574dd-4362-4e57-8dc0-b483d339f838\") " Sep 11 19:09:59.651338 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.651162 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d574dd-4362-4e57-8dc0-b483d339f838-plugins-config-volume\") pod \"a8d574dd-4362-4e57-8dc0-b483d339f838\" (UID: \"a8d574dd-4362-4e57-8dc0-b483d339f838\") " Sep 11 19:09:59.651338 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.651195 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6eaa7b-901d-4bf0-86ff-672169131846-plugins-config-volume\") pod \"bc6eaa7b-901d-4bf0-86ff-672169131846\" (UID: \"bc6eaa7b-901d-4bf0-86ff-672169131846\") " Sep 11 19:09:59.651338 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.651323 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4m7r\" (UniqueName: \"kubernetes.io/projected/bc6eaa7b-901d-4bf0-86ff-672169131846-kube-api-access-q4m7r\") pod \"bc6eaa7b-901d-4bf0-86ff-672169131846\" (UID: \"bc6eaa7b-901d-4bf0-86ff-672169131846\") " Sep 11 19:09:59.651806 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.651775 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6eaa7b-901d-4bf0-86ff-672169131846-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc6eaa7b-901d-4bf0-86ff-672169131846" (UID: "bc6eaa7b-901d-4bf0-86ff-672169131846"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:09:59.652189 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.652166 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d574dd-4362-4e57-8dc0-b483d339f838-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8d574dd-4362-4e57-8dc0-b483d339f838" (UID: "a8d574dd-4362-4e57-8dc0-b483d339f838"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:09:59.653855 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.653829 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d574dd-4362-4e57-8dc0-b483d339f838-kube-api-access-f97nl" (OuterVolumeSpecName: "kube-api-access-f97nl") pod "a8d574dd-4362-4e57-8dc0-b483d339f838" (UID: "a8d574dd-4362-4e57-8dc0-b483d339f838"). InnerVolumeSpecName "kube-api-access-f97nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:09:59.654145 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.654119 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6eaa7b-901d-4bf0-86ff-672169131846-kube-api-access-q4m7r" (OuterVolumeSpecName: "kube-api-access-q4m7r") pod "bc6eaa7b-901d-4bf0-86ff-672169131846" (UID: "bc6eaa7b-901d-4bf0-86ff-672169131846"). InnerVolumeSpecName "kube-api-access-q4m7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:09:59.752848 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.752761 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4m7r\" (UniqueName: \"kubernetes.io/projected/bc6eaa7b-901d-4bf0-86ff-672169131846-kube-api-access-q4m7r\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:09:59.752848 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.752796 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f97nl\" (UniqueName: \"kubernetes.io/projected/a8d574dd-4362-4e57-8dc0-b483d339f838-kube-api-access-f97nl\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:09:59.752848 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.752811 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d574dd-4362-4e57-8dc0-b483d339f838-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:09:59.752848 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.752825 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6eaa7b-901d-4bf0-86ff-672169131846-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:09:59.901432 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.901399 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:09:59.914249 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.914222 2556 generic.go:356] "Generic (PLEG): container finished" podID="bc6eaa7b-901d-4bf0-86ff-672169131846" containerID="f667eaf4bddb401042ed573021fb4e4285fbf6c72251a10299ec39606deb843d" exitCode=0 Sep 11 19:09:59.914393 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.914260 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" Sep 11 19:09:59.914393 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.914308 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" event={"ID":"bc6eaa7b-901d-4bf0-86ff-672169131846","Type":"ContainerDied","Data":"f667eaf4bddb401042ed573021fb4e4285fbf6c72251a10299ec39606deb843d"} Sep 11 19:09:59.914393 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.914356 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl" event={"ID":"bc6eaa7b-901d-4bf0-86ff-672169131846","Type":"ContainerDied","Data":"3480538f17f680f84c494c2cd222acc8975837d37d5cf133781b45186a9fd27d"} Sep 11 19:09:59.914393 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.914378 2556 scope.go:117] "RemoveContainer" containerID="f667eaf4bddb401042ed573021fb4e4285fbf6c72251a10299ec39606deb843d" Sep 11 19:09:59.915872 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.915848 2556 generic.go:356] "Generic (PLEG): container finished" podID="a8d574dd-4362-4e57-8dc0-b483d339f838" containerID="47e5d81353c8b3191583fdc01420acb7ace059941d0345194c870da81d1704ae" exitCode=0 Sep 11 19:09:59.916008 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.915891 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" Sep 11 19:09:59.916008 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.915942 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" event={"ID":"a8d574dd-4362-4e57-8dc0-b483d339f838","Type":"ContainerDied","Data":"47e5d81353c8b3191583fdc01420acb7ace059941d0345194c870da81d1704ae"} Sep 11 19:09:59.916008 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.915989 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f" event={"ID":"a8d574dd-4362-4e57-8dc0-b483d339f838","Type":"ContainerDied","Data":"81a7c46448a42c05cdb1606155d6946ad2062bb0f73984058cd06e06b8ab16c0"} Sep 11 19:09:59.924561 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.924523 2556 scope.go:117] "RemoveContainer" containerID="f667eaf4bddb401042ed573021fb4e4285fbf6c72251a10299ec39606deb843d" Sep 11 19:09:59.924884 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:09:59.924859 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f667eaf4bddb401042ed573021fb4e4285fbf6c72251a10299ec39606deb843d\": container with ID starting with f667eaf4bddb401042ed573021fb4e4285fbf6c72251a10299ec39606deb843d not found: ID does not exist" containerID="f667eaf4bddb401042ed573021fb4e4285fbf6c72251a10299ec39606deb843d" Sep 11 19:09:59.924945 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.924896 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f667eaf4bddb401042ed573021fb4e4285fbf6c72251a10299ec39606deb843d"} err="failed to get container status \"f667eaf4bddb401042ed573021fb4e4285fbf6c72251a10299ec39606deb843d\": rpc error: code = NotFound desc = could not find container \"f667eaf4bddb401042ed573021fb4e4285fbf6c72251a10299ec39606deb843d\": container with ID starting with f667eaf4bddb401042ed573021fb4e4285fbf6c72251a10299ec39606deb843d not found: ID does not exist" Sep 11 19:09:59.924945 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.924921 2556 scope.go:117] "RemoveContainer" containerID="47e5d81353c8b3191583fdc01420acb7ace059941d0345194c870da81d1704ae" Sep 11 19:09:59.934211 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.934193 2556 scope.go:117] "RemoveContainer" containerID="47e5d81353c8b3191583fdc01420acb7ace059941d0345194c870da81d1704ae" Sep 11 19:09:59.934482 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:09:59.934465 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e5d81353c8b3191583fdc01420acb7ace059941d0345194c870da81d1704ae\": container with ID starting with 47e5d81353c8b3191583fdc01420acb7ace059941d0345194c870da81d1704ae not found: ID does not exist" containerID="47e5d81353c8b3191583fdc01420acb7ace059941d0345194c870da81d1704ae" Sep 11 19:09:59.934572 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.934487 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e5d81353c8b3191583fdc01420acb7ace059941d0345194c870da81d1704ae"} err="failed to get container status \"47e5d81353c8b3191583fdc01420acb7ace059941d0345194c870da81d1704ae\": rpc error: code = NotFound desc = could not find container \"47e5d81353c8b3191583fdc01420acb7ace059941d0345194c870da81d1704ae\": container with ID starting with 47e5d81353c8b3191583fdc01420acb7ace059941d0345194c870da81d1704ae not found: ID does not exist" Sep 11 19:09:59.940387 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.940362 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl"] Sep 11 19:09:59.942557 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.942528 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-62499f-d965645b6-cd4vl"] Sep 11 19:09:59.953897 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.953875 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f"] Sep 11 19:09:59.966899 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.966872 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-62499f-84fdbfb547-mjj5f"] Sep 11 19:09:59.979222 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:09:59.979193 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:01.053623 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:01.053585 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:01.173508 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:01.173467 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8d574dd-4362-4e57-8dc0-b483d339f838" path="/var/lib/kubelet/pods/a8d574dd-4362-4e57-8dc0-b483d339f838/volumes" Sep 11 19:10:01.174030 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:01.174009 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc6eaa7b-901d-4bf0-86ff-672169131846" path="/var/lib/kubelet/pods/bc6eaa7b-901d-4bf0-86ff-672169131846/volumes" Sep 11 19:10:01.954780 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:01.954737 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:03.083909 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:03.083873 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:03.896533 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:03.896499 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:04.724689 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:04.724650 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:05.692567 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:05.692507 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:06.470064 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:06.470031 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:06.567869 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:06.567836 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:06.791549 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:06.791517 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:14.769567 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:14.769522 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:31.412985 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:31.412948 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:32.499927 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:32.499889 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:32.716340 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:32.716280 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:32.818719 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:32.818684 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:33.529013 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:33.528978 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:34.309575 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:34.309497 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:34.615126 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:34.615052 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:35.584386 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:35.584352 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:36.456106 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:36.456072 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:37.495349 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:37.495266 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:46.161183 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.161146 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq"] Sep 11 19:10:46.162059 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.162026 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" podUID="4846292a-38b1-4749-a23e-8f3a2860193a" containerName="maas-api" containerID="cri-o://1bfb84910201c1242fc39a007ba5475d1aced344b5a2d7c1d8e56da441f95769" gracePeriod=30 Sep 11 19:10:46.196111 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.196080 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:46.278282 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.278246 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:46.378615 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.378564 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:46.431526 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.431502 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:10:46.468397 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.468364 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:46.512555 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.512521 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4846292a-38b1-4749-a23e-8f3a2860193a-metrics-certs\") pod \"4846292a-38b1-4749-a23e-8f3a2860193a\" (UID: \"4846292a-38b1-4749-a23e-8f3a2860193a\") " Sep 11 19:10:46.512744 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.512594 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4846292a-38b1-4749-a23e-8f3a2860193a-openshift-service-ca\") pod \"4846292a-38b1-4749-a23e-8f3a2860193a\" (UID: \"4846292a-38b1-4749-a23e-8f3a2860193a\") " Sep 11 19:10:46.512744 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.512652 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4846292a-38b1-4749-a23e-8f3a2860193a-maas-api-tls\") pod \"4846292a-38b1-4749-a23e-8f3a2860193a\" (UID: \"4846292a-38b1-4749-a23e-8f3a2860193a\") " Sep 11 19:10:46.512744 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.512672 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mccx\" (UniqueName: \"kubernetes.io/projected/4846292a-38b1-4749-a23e-8f3a2860193a-kube-api-access-2mccx\") pod \"4846292a-38b1-4749-a23e-8f3a2860193a\" (UID: \"4846292a-38b1-4749-a23e-8f3a2860193a\") " Sep 11 19:10:46.513058 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.513027 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4846292a-38b1-4749-a23e-8f3a2860193a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4846292a-38b1-4749-a23e-8f3a2860193a" (UID: "4846292a-38b1-4749-a23e-8f3a2860193a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:10:46.514913 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.514885 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4846292a-38b1-4749-a23e-8f3a2860193a-kube-api-access-2mccx" (OuterVolumeSpecName: "kube-api-access-2mccx") pod "4846292a-38b1-4749-a23e-8f3a2860193a" (UID: "4846292a-38b1-4749-a23e-8f3a2860193a"). InnerVolumeSpecName "kube-api-access-2mccx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:10:46.515021 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.514933 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4846292a-38b1-4749-a23e-8f3a2860193a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4846292a-38b1-4749-a23e-8f3a2860193a" (UID: "4846292a-38b1-4749-a23e-8f3a2860193a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:10:46.515021 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.514969 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4846292a-38b1-4749-a23e-8f3a2860193a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4846292a-38b1-4749-a23e-8f3a2860193a" (UID: "4846292a-38b1-4749-a23e-8f3a2860193a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:10:46.613782 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.613751 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4846292a-38b1-4749-a23e-8f3a2860193a-metrics-certs\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:10:46.613782 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.613779 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4846292a-38b1-4749-a23e-8f3a2860193a-openshift-service-ca\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:10:46.613782 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.613788 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4846292a-38b1-4749-a23e-8f3a2860193a-maas-api-tls\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:10:46.614067 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.613797 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mccx\" (UniqueName: \"kubernetes.io/projected/4846292a-38b1-4749-a23e-8f3a2860193a-kube-api-access-2mccx\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:10:46.834504 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:46.834467 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:47.180871 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:47.180844 2556 generic.go:356] "Generic (PLEG): container finished" podID="4846292a-38b1-4749-a23e-8f3a2860193a" containerID="1bfb84910201c1242fc39a007ba5475d1aced344b5a2d7c1d8e56da441f95769" exitCode=0 Sep 11 19:10:47.181247 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:47.180895 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" Sep 11 19:10:47.181247 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:47.180921 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" event={"ID":"4846292a-38b1-4749-a23e-8f3a2860193a","Type":"ContainerDied","Data":"1bfb84910201c1242fc39a007ba5475d1aced344b5a2d7c1d8e56da441f95769"} Sep 11 19:10:47.181247 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:47.180964 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq" event={"ID":"4846292a-38b1-4749-a23e-8f3a2860193a","Type":"ContainerDied","Data":"d8c8f4b5a17d169b02be901ccbd45ed4d67337123e23f31b05567f6dd4bf834e"} Sep 11 19:10:47.181247 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:47.180987 2556 scope.go:117] "RemoveContainer" containerID="1bfb84910201c1242fc39a007ba5475d1aced344b5a2d7c1d8e56da441f95769" Sep 11 19:10:47.189683 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:47.189667 2556 scope.go:117] "RemoveContainer" containerID="1bfb84910201c1242fc39a007ba5475d1aced344b5a2d7c1d8e56da441f95769" Sep 11 19:10:47.189964 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:10:47.189946 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bfb84910201c1242fc39a007ba5475d1aced344b5a2d7c1d8e56da441f95769\": container with ID starting with 1bfb84910201c1242fc39a007ba5475d1aced344b5a2d7c1d8e56da441f95769 not found: ID does not exist" containerID="1bfb84910201c1242fc39a007ba5475d1aced344b5a2d7c1d8e56da441f95769" Sep 11 19:10:47.190009 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:47.189973 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfb84910201c1242fc39a007ba5475d1aced344b5a2d7c1d8e56da441f95769"} err="failed to get container status \"1bfb84910201c1242fc39a007ba5475d1aced344b5a2d7c1d8e56da441f95769\": rpc error: code = NotFound desc = could not find container \"1bfb84910201c1242fc39a007ba5475d1aced344b5a2d7c1d8e56da441f95769\": container with ID starting with 1bfb84910201c1242fc39a007ba5475d1aced344b5a2d7c1d8e56da441f95769 not found: ID does not exist" Sep 11 19:10:47.204581 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:47.204554 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq"] Sep 11 19:10:47.212427 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:47.212403 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ad53a9d-65cd5dfd76-lwlzq"] Sep 11 19:10:47.906132 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:47.906096 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:48.793216 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:48.793182 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:49.172355 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:49.172277 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4846292a-38b1-4749-a23e-8f3a2860193a" path="/var/lib/kubelet/pods/4846292a-38b1-4749-a23e-8f3a2860193a/volumes" Sep 11 19:10:49.602943 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:49.602909 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:49.703670 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:49.703634 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:50.584927 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:50.584895 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:51.468618 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:51.468581 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:10:54.671528 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:10:54.671495 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:01.073376 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:01.073340 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:05.960605 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:05.960568 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:08.355754 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:08.355712 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:08.552509 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:08.552460 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:29.673300 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:29.673263 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:31.975689 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:31.975649 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:32.592356 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:32.592320 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:38.764330 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:38.764301 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:38.865758 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:38.865691 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:39.041840 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:39.041802 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:40.349490 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:40.349440 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:40.555432 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:40.555387 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:41.142284 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:41.142244 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:41.238502 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:41.238465 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:42.587220 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:42.587185 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:48.751846 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:48.751805 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:48.868750 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:48.868715 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:49.409756 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:49.409726 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:56.242558 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:56.242515 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:56.348048 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:56.348012 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:56.449323 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:56.449289 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:59.436125 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:59.436087 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:11:59.536868 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:11:59.536833 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:03.734407 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:03.734369 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:06.942755 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:06.942714 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:07.164613 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:07.164510 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:15.266089 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:15.266056 2556 scope.go:117] "RemoveContainer" containerID="5ea14a4134184d5c011e8244e48f39b9c1dccf6cf0954da7548e6ef85ecbeaee" Sep 11 19:12:24.699560 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:24.699499 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:28.097497 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:28.097462 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:35.045771 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:35.045737 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:35.144926 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:35.144892 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:35.411936 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:35.411843 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:35.763635 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:35.763527 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:35.864359 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:35.864323 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:38.388861 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:38.388800 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:39.007370 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:39.007334 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:39.172775 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:39.172745 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:39.402057 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:39.402031 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:40.302099 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:40.302065 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:41.197725 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:41.197690 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:42.042882 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:42.042846 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:42.900401 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:42.900363 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:44.044230 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:44.044198 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:44.144887 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:44.144846 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:44.492560 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:44.492457 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:44.670273 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:44.670241 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:44.772413 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:44.772328 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:55.867531 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:55.867493 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:12:55.940191 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:12:55.940154 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:05.712888 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:05.712856 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:05.812842 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:05.812807 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:06.599110 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:06.599073 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:07.673755 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:07.673720 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:08.570427 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:08.570393 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:09.143442 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:09.143404 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:09.504333 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:09.504237 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:09.632356 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:09.632324 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:09.966198 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:09.966164 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:10.249644 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:10.249559 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:10.682111 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:10.682083 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:11.510207 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:11.510174 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:12.499315 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:12.499281 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:13.203228 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:13.203190 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:13.300861 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:13.300828 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:13.533686 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:13.533655 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:14.713518 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:14.713480 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:14.817080 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:14.817047 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:14.917686 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:14.917662 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:15.120958 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.120924 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 19:13:15.124773 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.124744 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 19:13:15.259862 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.259827 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:15.286282 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.286257 2556 scope.go:117] "RemoveContainer" containerID="3128023bc5d6662026693158fc29a78b8cfde76b1dba2e464514dd47073bbec1" Sep 11 19:13:15.355762 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.355726 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:15.387508 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.387416 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf"] Sep 11 19:13:15.388072 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.388052 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8d574dd-4362-4e57-8dc0-b483d339f838" containerName="payload-processing" Sep 11 19:13:15.388072 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.388073 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8d574dd-4362-4e57-8dc0-b483d339f838" containerName="payload-processing" Sep 11 19:13:15.388201 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.388081 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc6eaa7b-901d-4bf0-86ff-672169131846" containerName="payload-pre-processing" Sep 11 19:13:15.388201 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.388088 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc6eaa7b-901d-4bf0-86ff-672169131846" containerName="payload-pre-processing" Sep 11 19:13:15.388201 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.388108 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4846292a-38b1-4749-a23e-8f3a2860193a" containerName="maas-api" Sep 11 19:13:15.388201 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.388115 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4846292a-38b1-4749-a23e-8f3a2860193a" containerName="maas-api" Sep 11 19:13:15.388201 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.388177 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc6eaa7b-901d-4bf0-86ff-672169131846" containerName="payload-pre-processing" Sep 11 19:13:15.388201 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.388190 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8d574dd-4362-4e57-8dc0-b483d339f838" containerName="payload-processing" Sep 11 19:13:15.388201 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.388197 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4846292a-38b1-4749-a23e-8f3a2860193a" containerName="maas-api" Sep 11 19:13:15.392712 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.392696 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" Sep 11 19:13:15.395118 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.395095 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-852059\"" Sep 11 19:13:15.402505 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.402479 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf"] Sep 11 19:13:15.499998 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.499960 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm98w\" (UniqueName: \"kubernetes.io/projected/85d664d0-b623-4325-ae66-f577a10db529-kube-api-access-xm98w\") pod \"payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf\" (UID: \"85d664d0-b623-4325-ae66-f577a10db529\") " pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" Sep 11 19:13:15.500160 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.500016 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85d664d0-b623-4325-ae66-f577a10db529-plugins-config-volume\") pod \"payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf\" (UID: \"85d664d0-b623-4325-ae66-f577a10db529\") " pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" Sep 11 19:13:15.601316 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.601280 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm98w\" (UniqueName: \"kubernetes.io/projected/85d664d0-b623-4325-ae66-f577a10db529-kube-api-access-xm98w\") pod \"payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf\" (UID: \"85d664d0-b623-4325-ae66-f577a10db529\") " pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" Sep 11 19:13:15.601462 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.601325 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85d664d0-b623-4325-ae66-f577a10db529-plugins-config-volume\") pod \"payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf\" (UID: \"85d664d0-b623-4325-ae66-f577a10db529\") " pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" Sep 11 19:13:15.601998 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.601980 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85d664d0-b623-4325-ae66-f577a10db529-plugins-config-volume\") pod \"payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf\" (UID: \"85d664d0-b623-4325-ae66-f577a10db529\") " pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" Sep 11 19:13:15.610113 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.610083 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm98w\" (UniqueName: \"kubernetes.io/projected/85d664d0-b623-4325-ae66-f577a10db529-kube-api-access-xm98w\") pod \"payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf\" (UID: \"85d664d0-b623-4325-ae66-f577a10db529\") " pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" Sep 11 19:13:15.704888 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.704807 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" Sep 11 19:13:15.718529 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.718504 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:15.982738 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.982713 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf"] Sep 11 19:13:15.984803 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:13:15.984778 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d664d0_b623_4325_ae66_f577a10db529.slice/crio-12872f5575d55a898175c7c35fa46ac3181ec60a4b605996f7c5c3ca1b30079a WatchSource:0}: Error finding container 12872f5575d55a898175c7c35fa46ac3181ec60a4b605996f7c5c3ca1b30079a: Status 404 returned error can't find the container with id 12872f5575d55a898175c7c35fa46ac3181ec60a4b605996f7c5c3ca1b30079a Sep 11 19:13:15.994650 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:15.994528 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" event={"ID":"85d664d0-b623-4325-ae66-f577a10db529","Type":"ContainerStarted","Data":"12872f5575d55a898175c7c35fa46ac3181ec60a4b605996f7c5c3ca1b30079a"} Sep 11 19:13:16.581447 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:16.581413 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:17.000860 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:17.000766 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" event={"ID":"85d664d0-b623-4325-ae66-f577a10db529","Type":"ContainerStarted","Data":"956fefb36208700dbc96563e0a64d25934c9e329015bd5d0b97790fa212d4786"} Sep 11 19:13:17.000860 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:17.000825 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" Sep 11 19:13:17.021313 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:17.021265 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" podStartSLOduration=2.021250293 podStartE2EDuration="2.021250293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:13:17.018618446 +0000 UTC m=+1502.472424428" watchObservedRunningTime="2026-09-11 19:13:17.021250293 +0000 UTC m=+1502.475056248" Sep 11 19:13:17.505264 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:17.505227 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:18.464532 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:18.464491 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:19.281585 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:19.281535 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:20.059069 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:20.059026 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:28.008387 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:28.008349 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" Sep 11 19:13:37.411437 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:37.411346 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:38.124556 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:38.124513 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:38.270213 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:38.270185 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:39.072775 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:39.072660 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:39.979843 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:39.979810 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:40.847046 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:40.847010 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:41.838639 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:41.838596 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:42.759845 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:42.759799 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:43.607654 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:43.607611 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:44.497465 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:44.497433 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:45.626407 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:45.626370 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:46.576224 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:46.576185 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:47.431846 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:47.431817 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:48.477926 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:48.477883 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:51.384506 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:51.384468 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:57.022303 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:57.022269 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:57.094021 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:57.093976 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:57.178934 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:57.178887 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:57.262135 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:57.262102 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:57.328771 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:57.328739 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:57.613604 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:57.613500 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:57.767349 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:57.767312 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:58.608984 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:58.608921 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:13:59.459448 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:13:59.459408 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:00.310161 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:00.310125 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:01.035082 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:01.035047 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:01.967884 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:01.967853 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:02.696260 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:02.696230 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:02.797196 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:02.797157 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:03.549014 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:03.548971 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:03.650742 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:03.650711 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:03.768648 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:03.768562 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:03.867166 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:03.867081 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:04.585511 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:04.585469 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:05.325314 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:05.325269 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:05.425331 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:05.425299 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:06.105613 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:06.105579 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:07.645524 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:07.645488 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:07.746487 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:07.746453 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:07.865370 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:07.865333 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:08.658722 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:08.658690 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:09.489258 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:09.489226 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:14.258123 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:14.258089 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:18.574426 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.574385 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:18.598799 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.598768 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9"] Sep 11 19:14:18.599010 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.598968 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" podUID="3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55" containerName="payload-processing" containerID="cri-o://1789bdd064f3fd2dc2ce944aa5b8682f1708eca1b0ccf31771d82e40a95e8169" gracePeriod=30 Sep 11 19:14:18.604141 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.604019 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv"] Sep 11 19:14:18.604784 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.604606 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" podUID="b6a333da-fe91-4593-b69e-f49b76efba30" containerName="payload-pre-processing" containerID="cri-o://fc88a4071bb59abcd9490331390b0c7a1976d5e7ab293d2cfedeef6be265a5fe" gracePeriod=30 Sep 11 19:14:18.650857 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.650815 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:18.730732 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.730632 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:18.804312 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.804263 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:18.864936 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.864911 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" Sep 11 19:14:18.875683 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.875653 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:18.896385 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.896361 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" Sep 11 19:14:18.917683 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.917658 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a333da-fe91-4593-b69e-f49b76efba30-plugins-config-volume\") pod \"b6a333da-fe91-4593-b69e-f49b76efba30\" (UID: \"b6a333da-fe91-4593-b69e-f49b76efba30\") " Sep 11 19:14:18.917873 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.917749 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55-plugins-config-volume\") pod \"3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55\" (UID: \"3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55\") " Sep 11 19:14:18.917873 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.917797 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps46k\" (UniqueName: \"kubernetes.io/projected/3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55-kube-api-access-ps46k\") pod \"3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55\" (UID: \"3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55\") " Sep 11 19:14:18.917873 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.917822 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtw64\" (UniqueName: \"kubernetes.io/projected/b6a333da-fe91-4593-b69e-f49b76efba30-kube-api-access-dtw64\") pod \"b6a333da-fe91-4593-b69e-f49b76efba30\" (UID: \"b6a333da-fe91-4593-b69e-f49b76efba30\") " Sep 11 19:14:18.918139 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.918108 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a333da-fe91-4593-b69e-f49b76efba30-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6a333da-fe91-4593-b69e-f49b76efba30" (UID: "b6a333da-fe91-4593-b69e-f49b76efba30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:14:18.918271 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.918153 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55" (UID: "3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:14:18.920125 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.920090 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55-kube-api-access-ps46k" (OuterVolumeSpecName: "kube-api-access-ps46k") pod "3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55" (UID: "3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55"). InnerVolumeSpecName "kube-api-access-ps46k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:14:18.920239 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:18.920158 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a333da-fe91-4593-b69e-f49b76efba30-kube-api-access-dtw64" (OuterVolumeSpecName: "kube-api-access-dtw64") pod "b6a333da-fe91-4593-b69e-f49b76efba30" (UID: "b6a333da-fe91-4593-b69e-f49b76efba30"). InnerVolumeSpecName "kube-api-access-dtw64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:14:19.018607 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.018571 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a333da-fe91-4593-b69e-f49b76efba30-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:19.018607 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.018605 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:19.018607 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.018615 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps46k\" (UniqueName: \"kubernetes.io/projected/3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55-kube-api-access-ps46k\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:19.018860 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.018623 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtw64\" (UniqueName: \"kubernetes.io/projected/b6a333da-fe91-4593-b69e-f49b76efba30-kube-api-access-dtw64\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:19.155358 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.155262 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:19.253150 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.253111 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:19.340817 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.340782 2556 generic.go:356] "Generic (PLEG): container finished" podID="b6a333da-fe91-4593-b69e-f49b76efba30" containerID="fc88a4071bb59abcd9490331390b0c7a1976d5e7ab293d2cfedeef6be265a5fe" exitCode=0 Sep 11 19:14:19.341024 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.340823 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" Sep 11 19:14:19.341024 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.340868 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" event={"ID":"b6a333da-fe91-4593-b69e-f49b76efba30","Type":"ContainerDied","Data":"fc88a4071bb59abcd9490331390b0c7a1976d5e7ab293d2cfedeef6be265a5fe"} Sep 11 19:14:19.341024 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.340918 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv" event={"ID":"b6a333da-fe91-4593-b69e-f49b76efba30","Type":"ContainerDied","Data":"caac8f512924b0e390dd25e88b7cc04a086b8f7c37eef3258b1a3e3a261474f6"} Sep 11 19:14:19.341024 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.340942 2556 scope.go:117] "RemoveContainer" containerID="fc88a4071bb59abcd9490331390b0c7a1976d5e7ab293d2cfedeef6be265a5fe" Sep 11 19:14:19.342297 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.342223 2556 generic.go:356] "Generic (PLEG): container finished" podID="3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55" containerID="1789bdd064f3fd2dc2ce944aa5b8682f1708eca1b0ccf31771d82e40a95e8169" exitCode=0 Sep 11 19:14:19.342297 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.342264 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" Sep 11 19:14:19.342426 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.342310 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" event={"ID":"3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55","Type":"ContainerDied","Data":"1789bdd064f3fd2dc2ce944aa5b8682f1708eca1b0ccf31771d82e40a95e8169"} Sep 11 19:14:19.342426 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.342361 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9" event={"ID":"3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55","Type":"ContainerDied","Data":"00906e80a5722be4c5f0321e54927f38d5432a3b3ea3a10a7dc756ca9233d999"} Sep 11 19:14:19.353379 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.353357 2556 scope.go:117] "RemoveContainer" containerID="fc88a4071bb59abcd9490331390b0c7a1976d5e7ab293d2cfedeef6be265a5fe" Sep 11 19:14:19.353725 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:14:19.353707 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc88a4071bb59abcd9490331390b0c7a1976d5e7ab293d2cfedeef6be265a5fe\": container with ID starting with fc88a4071bb59abcd9490331390b0c7a1976d5e7ab293d2cfedeef6be265a5fe not found: ID does not exist" containerID="fc88a4071bb59abcd9490331390b0c7a1976d5e7ab293d2cfedeef6be265a5fe" Sep 11 19:14:19.353820 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.353734 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc88a4071bb59abcd9490331390b0c7a1976d5e7ab293d2cfedeef6be265a5fe"} err="failed to get container status \"fc88a4071bb59abcd9490331390b0c7a1976d5e7ab293d2cfedeef6be265a5fe\": rpc error: code = NotFound desc = could not find container \"fc88a4071bb59abcd9490331390b0c7a1976d5e7ab293d2cfedeef6be265a5fe\": container with ID starting with fc88a4071bb59abcd9490331390b0c7a1976d5e7ab293d2cfedeef6be265a5fe not found: ID does not exist" Sep 11 19:14:19.353820 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.353751 2556 scope.go:117] "RemoveContainer" containerID="1789bdd064f3fd2dc2ce944aa5b8682f1708eca1b0ccf31771d82e40a95e8169" Sep 11 19:14:19.361835 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.361821 2556 scope.go:117] "RemoveContainer" containerID="1789bdd064f3fd2dc2ce944aa5b8682f1708eca1b0ccf31771d82e40a95e8169" Sep 11 19:14:19.362110 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:14:19.362085 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1789bdd064f3fd2dc2ce944aa5b8682f1708eca1b0ccf31771d82e40a95e8169\": container with ID starting with 1789bdd064f3fd2dc2ce944aa5b8682f1708eca1b0ccf31771d82e40a95e8169 not found: ID does not exist" containerID="1789bdd064f3fd2dc2ce944aa5b8682f1708eca1b0ccf31771d82e40a95e8169" Sep 11 19:14:19.362171 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.362112 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1789bdd064f3fd2dc2ce944aa5b8682f1708eca1b0ccf31771d82e40a95e8169"} err="failed to get container status \"1789bdd064f3fd2dc2ce944aa5b8682f1708eca1b0ccf31771d82e40a95e8169\": rpc error: code = NotFound desc = could not find container \"1789bdd064f3fd2dc2ce944aa5b8682f1708eca1b0ccf31771d82e40a95e8169\": container with ID starting with 1789bdd064f3fd2dc2ce944aa5b8682f1708eca1b0ccf31771d82e40a95e8169 not found: ID does not exist" Sep 11 19:14:19.370043 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.370021 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv"] Sep 11 19:14:19.385195 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.385173 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c1aebc-744d4cb8b6-jvmwv"] Sep 11 19:14:19.411040 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.410980 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9"] Sep 11 19:14:19.415917 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:19.415892 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c1aebc-67f6f6bbb9-lqhg9"] Sep 11 19:14:20.304032 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:20.303989 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:21.098765 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:21.098731 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:21.176475 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:21.176431 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55" path="/var/lib/kubelet/pods/3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55/volumes" Sep 11 19:14:21.176934 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:21.176913 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6a333da-fe91-4593-b69e-f49b76efba30" path="/var/lib/kubelet/pods/b6a333da-fe91-4593-b69e-f49b76efba30/volumes" Sep 11 19:14:21.905091 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:21.905061 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:21.930647 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:21.930613 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf"] Sep 11 19:14:21.930869 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:21.930839 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" podUID="85d664d0-b623-4325-ae66-f577a10db529" containerName="payload-processing" containerID="cri-o://956fefb36208700dbc96563e0a64d25934c9e329015bd5d0b97790fa212d4786" gracePeriod=30 Sep 11 19:14:22.005397 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.005362 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:22.105260 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.105226 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:22.168357 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.168271 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:22.196983 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.196955 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" Sep 11 19:14:22.248790 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.248753 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85d664d0-b623-4325-ae66-f577a10db529-plugins-config-volume\") pod \"85d664d0-b623-4325-ae66-f577a10db529\" (UID: \"85d664d0-b623-4325-ae66-f577a10db529\") " Sep 11 19:14:22.248992 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.248861 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm98w\" (UniqueName: \"kubernetes.io/projected/85d664d0-b623-4325-ae66-f577a10db529-kube-api-access-xm98w\") pod \"85d664d0-b623-4325-ae66-f577a10db529\" (UID: \"85d664d0-b623-4325-ae66-f577a10db529\") " Sep 11 19:14:22.249166 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.249139 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d664d0-b623-4325-ae66-f577a10db529-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85d664d0-b623-4325-ae66-f577a10db529" (UID: "85d664d0-b623-4325-ae66-f577a10db529"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:14:22.251057 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.251027 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d664d0-b623-4325-ae66-f577a10db529-kube-api-access-xm98w" (OuterVolumeSpecName: "kube-api-access-xm98w") pod "85d664d0-b623-4325-ae66-f577a10db529" (UID: "85d664d0-b623-4325-ae66-f577a10db529"). InnerVolumeSpecName "kube-api-access-xm98w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:14:22.330448 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.330417 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:22.349975 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.349931 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85d664d0-b623-4325-ae66-f577a10db529-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:22.349975 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.349973 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm98w\" (UniqueName: \"kubernetes.io/projected/85d664d0-b623-4325-ae66-f577a10db529-kube-api-access-xm98w\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:22.366708 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.366674 2556 generic.go:356] "Generic (PLEG): container finished" podID="85d664d0-b623-4325-ae66-f577a10db529" containerID="956fefb36208700dbc96563e0a64d25934c9e329015bd5d0b97790fa212d4786" exitCode=0 Sep 11 19:14:22.366849 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.366723 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" Sep 11 19:14:22.366849 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.366743 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" event={"ID":"85d664d0-b623-4325-ae66-f577a10db529","Type":"ContainerDied","Data":"956fefb36208700dbc96563e0a64d25934c9e329015bd5d0b97790fa212d4786"} Sep 11 19:14:22.366849 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.366786 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf" event={"ID":"85d664d0-b623-4325-ae66-f577a10db529","Type":"ContainerDied","Data":"12872f5575d55a898175c7c35fa46ac3181ec60a4b605996f7c5c3ca1b30079a"} Sep 11 19:14:22.366849 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.366806 2556 scope.go:117] "RemoveContainer" containerID="956fefb36208700dbc96563e0a64d25934c9e329015bd5d0b97790fa212d4786" Sep 11 19:14:22.376335 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.376313 2556 scope.go:117] "RemoveContainer" containerID="956fefb36208700dbc96563e0a64d25934c9e329015bd5d0b97790fa212d4786" Sep 11 19:14:22.376709 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:14:22.376687 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"956fefb36208700dbc96563e0a64d25934c9e329015bd5d0b97790fa212d4786\": container with ID starting with 956fefb36208700dbc96563e0a64d25934c9e329015bd5d0b97790fa212d4786 not found: ID does not exist" containerID="956fefb36208700dbc96563e0a64d25934c9e329015bd5d0b97790fa212d4786" Sep 11 19:14:22.376773 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.376721 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956fefb36208700dbc96563e0a64d25934c9e329015bd5d0b97790fa212d4786"} err="failed to get container status \"956fefb36208700dbc96563e0a64d25934c9e329015bd5d0b97790fa212d4786\": rpc error: code = NotFound desc = could not find container \"956fefb36208700dbc96563e0a64d25934c9e329015bd5d0b97790fa212d4786\": container with ID starting with 956fefb36208700dbc96563e0a64d25934c9e329015bd5d0b97790fa212d4786 not found: ID does not exist" Sep 11 19:14:22.387138 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.387109 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf"] Sep 11 19:14:22.390906 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.390875 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-852059-77dc97fdc5-46kwf"] Sep 11 19:14:22.441064 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:22.440932 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:23.069057 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:23.069007 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:23.174277 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:23.174241 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85d664d0-b623-4325-ae66-f577a10db529" path="/var/lib/kubelet/pods/85d664d0-b623-4325-ae66-f577a10db529/volumes" Sep 11 19:14:23.884884 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:23.884849 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:24.584384 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:24.584338 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:25.196293 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:25.196258 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:25.352957 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:25.352917 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:25.443324 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:25.443280 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:25.528663 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:25.528535 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:26.268563 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:26.268508 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:28.100786 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:28.100751 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:28.158065 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:28.158038 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:28.339568 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:28.339510 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:29.357144 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:29.357104 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:30.170129 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:30.170099 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:30.863451 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:30.863415 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:31.528097 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:31.528054 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:31.729611 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:31.729580 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:32.439969 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:32.439931 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:34.338763 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:34.338733 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:34.833687 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:34.833654 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:39.128918 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.128886 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc"] Sep 11 19:14:39.129364 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.129125 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" podUID="ce37bbd2-eb10-4c58-91c5-fdf15fa56810" containerName="maas-api" containerID="cri-o://acfee418d17ce3a069a7efc5288605a271f8cbdc46aff8e94ed3ea816a03d872" gracePeriod=30 Sep 11 19:14:39.151810 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.151654 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:39.231598 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.231567 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq"] Sep 11 19:14:39.231850 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.231824 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" podUID="7e5dca55-4746-454a-8746-2c54d3f1fa60" containerName="payload-processing" containerID="cri-o://e2813982ad58650924b56211125feb627738b474251cdf6af0d9c9d3aae90495" gracePeriod=30 Sep 11 19:14:39.242287 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.242260 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp"] Sep 11 19:14:39.242741 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.242491 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" podUID="9d67ab2c-ee3a-448c-8374-16e1bc25e089" containerName="payload-pre-processing" containerID="cri-o://59f3a12f0e8c17fa26c4fc6533f8acefc12806d7a4fea3b89358a9a876fad517" gracePeriod=30 Sep 11 19:14:39.262200 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.262139 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:39.361313 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.361285 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:39.411310 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.411284 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:14:39.421959 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.421930 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:39.470513 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.470479 2556 generic.go:356] "Generic (PLEG): container finished" podID="7e5dca55-4746-454a-8746-2c54d3f1fa60" containerID="e2813982ad58650924b56211125feb627738b474251cdf6af0d9c9d3aae90495" exitCode=0 Sep 11 19:14:39.470697 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.470596 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" event={"ID":"7e5dca55-4746-454a-8746-2c54d3f1fa60","Type":"ContainerDied","Data":"e2813982ad58650924b56211125feb627738b474251cdf6af0d9c9d3aae90495"} Sep 11 19:14:39.472431 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.472402 2556 generic.go:356] "Generic (PLEG): container finished" podID="ce37bbd2-eb10-4c58-91c5-fdf15fa56810" containerID="acfee418d17ce3a069a7efc5288605a271f8cbdc46aff8e94ed3ea816a03d872" exitCode=0 Sep 11 19:14:39.472592 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.472472 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" Sep 11 19:14:39.472592 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.472488 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" event={"ID":"ce37bbd2-eb10-4c58-91c5-fdf15fa56810","Type":"ContainerDied","Data":"acfee418d17ce3a069a7efc5288605a271f8cbdc46aff8e94ed3ea816a03d872"} Sep 11 19:14:39.472592 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.472528 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc" event={"ID":"ce37bbd2-eb10-4c58-91c5-fdf15fa56810","Type":"ContainerDied","Data":"92271e37b1ef4879a500c708a4ebb71ec14ee7f6088a94cbaa914d74334135ff"} Sep 11 19:14:39.472592 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.472571 2556 scope.go:117] "RemoveContainer" containerID="acfee418d17ce3a069a7efc5288605a271f8cbdc46aff8e94ed3ea816a03d872" Sep 11 19:14:39.474997 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.474975 2556 generic.go:356] "Generic (PLEG): container finished" podID="9d67ab2c-ee3a-448c-8374-16e1bc25e089" containerID="59f3a12f0e8c17fa26c4fc6533f8acefc12806d7a4fea3b89358a9a876fad517" exitCode=0 Sep 11 19:14:39.475101 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.475054 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" event={"ID":"9d67ab2c-ee3a-448c-8374-16e1bc25e089","Type":"ContainerDied","Data":"59f3a12f0e8c17fa26c4fc6533f8acefc12806d7a4fea3b89358a9a876fad517"} Sep 11 19:14:39.487975 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.487954 2556 scope.go:117] "RemoveContainer" containerID="acfee418d17ce3a069a7efc5288605a271f8cbdc46aff8e94ed3ea816a03d872" Sep 11 19:14:39.488362 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:14:39.488333 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acfee418d17ce3a069a7efc5288605a271f8cbdc46aff8e94ed3ea816a03d872\": container with ID starting with acfee418d17ce3a069a7efc5288605a271f8cbdc46aff8e94ed3ea816a03d872 not found: ID does not exist" containerID="acfee418d17ce3a069a7efc5288605a271f8cbdc46aff8e94ed3ea816a03d872" Sep 11 19:14:39.488461 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.488372 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acfee418d17ce3a069a7efc5288605a271f8cbdc46aff8e94ed3ea816a03d872"} err="failed to get container status \"acfee418d17ce3a069a7efc5288605a271f8cbdc46aff8e94ed3ea816a03d872\": rpc error: code = NotFound desc = could not find container \"acfee418d17ce3a069a7efc5288605a271f8cbdc46aff8e94ed3ea816a03d872\": container with ID starting with acfee418d17ce3a069a7efc5288605a271f8cbdc46aff8e94ed3ea816a03d872 not found: ID does not exist" Sep 11 19:14:39.521678 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.521660 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" Sep 11 19:14:39.522006 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.521989 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-metrics-certs\") pod \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\" (UID: \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\") " Sep 11 19:14:39.522087 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.522047 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-openshift-service-ca\") pod \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\" (UID: \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\") " Sep 11 19:14:39.522146 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.522096 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-maas-api-tls\") pod \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\" (UID: \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\") " Sep 11 19:14:39.522198 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.522158 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlvn8\" (UniqueName: \"kubernetes.io/projected/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-kube-api-access-dlvn8\") pod \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\" (UID: \"ce37bbd2-eb10-4c58-91c5-fdf15fa56810\") " Sep 11 19:14:39.522411 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.522383 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ce37bbd2-eb10-4c58-91c5-fdf15fa56810" (UID: "ce37bbd2-eb10-4c58-91c5-fdf15fa56810"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:14:39.522513 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.522496 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-openshift-service-ca\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:39.524419 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.524398 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ce37bbd2-eb10-4c58-91c5-fdf15fa56810" (UID: "ce37bbd2-eb10-4c58-91c5-fdf15fa56810"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:14:39.524577 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.524562 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ce37bbd2-eb10-4c58-91c5-fdf15fa56810" (UID: "ce37bbd2-eb10-4c58-91c5-fdf15fa56810"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:14:39.524647 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.524587 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-kube-api-access-dlvn8" (OuterVolumeSpecName: "kube-api-access-dlvn8") pod "ce37bbd2-eb10-4c58-91c5-fdf15fa56810" (UID: "ce37bbd2-eb10-4c58-91c5-fdf15fa56810"). InnerVolumeSpecName "kube-api-access-dlvn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:14:39.526104 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.526088 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" Sep 11 19:14:39.623236 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.623202 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5dca55-4746-454a-8746-2c54d3f1fa60-plugins-config-volume\") pod \"7e5dca55-4746-454a-8746-2c54d3f1fa60\" (UID: \"7e5dca55-4746-454a-8746-2c54d3f1fa60\") " Sep 11 19:14:39.623429 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.623246 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d67ab2c-ee3a-448c-8374-16e1bc25e089-plugins-config-volume\") pod \"9d67ab2c-ee3a-448c-8374-16e1bc25e089\" (UID: \"9d67ab2c-ee3a-448c-8374-16e1bc25e089\") " Sep 11 19:14:39.623429 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.623277 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdv6h\" (UniqueName: \"kubernetes.io/projected/9d67ab2c-ee3a-448c-8374-16e1bc25e089-kube-api-access-rdv6h\") pod \"9d67ab2c-ee3a-448c-8374-16e1bc25e089\" (UID: \"9d67ab2c-ee3a-448c-8374-16e1bc25e089\") " Sep 11 19:14:39.623429 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.623357 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq5kl\" (UniqueName: \"kubernetes.io/projected/7e5dca55-4746-454a-8746-2c54d3f1fa60-kube-api-access-tq5kl\") pod \"7e5dca55-4746-454a-8746-2c54d3f1fa60\" (UID: \"7e5dca55-4746-454a-8746-2c54d3f1fa60\") " Sep 11 19:14:39.623707 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.623682 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5dca55-4746-454a-8746-2c54d3f1fa60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e5dca55-4746-454a-8746-2c54d3f1fa60" (UID: "7e5dca55-4746-454a-8746-2c54d3f1fa60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:14:39.623763 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.623682 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d67ab2c-ee3a-448c-8374-16e1bc25e089-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d67ab2c-ee3a-448c-8374-16e1bc25e089" (UID: "9d67ab2c-ee3a-448c-8374-16e1bc25e089"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:14:39.623805 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.623771 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-maas-api-tls\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:39.623805 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.623788 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlvn8\" (UniqueName: \"kubernetes.io/projected/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-kube-api-access-dlvn8\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:39.623805 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.623800 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5dca55-4746-454a-8746-2c54d3f1fa60-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:39.623903 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.623808 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d67ab2c-ee3a-448c-8374-16e1bc25e089-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:39.623903 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.623817 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce37bbd2-eb10-4c58-91c5-fdf15fa56810-metrics-certs\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:39.625386 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.625365 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5dca55-4746-454a-8746-2c54d3f1fa60-kube-api-access-tq5kl" (OuterVolumeSpecName: "kube-api-access-tq5kl") pod "7e5dca55-4746-454a-8746-2c54d3f1fa60" (UID: "7e5dca55-4746-454a-8746-2c54d3f1fa60"). InnerVolumeSpecName "kube-api-access-tq5kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:14:39.625434 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.625384 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d67ab2c-ee3a-448c-8374-16e1bc25e089-kube-api-access-rdv6h" (OuterVolumeSpecName: "kube-api-access-rdv6h") pod "9d67ab2c-ee3a-448c-8374-16e1bc25e089" (UID: "9d67ab2c-ee3a-448c-8374-16e1bc25e089"). InnerVolumeSpecName "kube-api-access-rdv6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:14:39.724797 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.724727 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq5kl\" (UniqueName: \"kubernetes.io/projected/7e5dca55-4746-454a-8746-2c54d3f1fa60-kube-api-access-tq5kl\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:39.724797 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.724753 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdv6h\" (UniqueName: \"kubernetes.io/projected/9d67ab2c-ee3a-448c-8374-16e1bc25e089-kube-api-access-rdv6h\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:39.747714 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.747683 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:39.794282 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.794243 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc"] Sep 11 19:14:39.796525 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.796500 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-19647e-699f94976-7p8hc"] Sep 11 19:14:39.853836 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:39.853795 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:40.482327 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:40.482245 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" Sep 11 19:14:40.482327 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:40.482273 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp" event={"ID":"9d67ab2c-ee3a-448c-8374-16e1bc25e089","Type":"ContainerDied","Data":"503b01b8642e28b228df2981dd278b365cb39816f4d6118814d241d44409dd84"} Sep 11 19:14:40.482327 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:40.482322 2556 scope.go:117] "RemoveContainer" containerID="59f3a12f0e8c17fa26c4fc6533f8acefc12806d7a4fea3b89358a9a876fad517" Sep 11 19:14:40.483706 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:40.483633 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" Sep 11 19:14:40.483706 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:40.483658 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq" event={"ID":"7e5dca55-4746-454a-8746-2c54d3f1fa60","Type":"ContainerDied","Data":"f5ccbc6a5d2eae8953a4a3028683a348da9f933d0bbb7af7363d233fdc5b66e6"} Sep 11 19:14:40.491453 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:40.491334 2556 scope.go:117] "RemoveContainer" containerID="e2813982ad58650924b56211125feb627738b474251cdf6af0d9c9d3aae90495" Sep 11 19:14:40.505406 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:40.505385 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq"] Sep 11 19:14:40.509321 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:40.509301 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-19647e-5597cc7b89-l26hq"] Sep 11 19:14:40.517770 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:40.517749 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp"] Sep 11 19:14:40.521348 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:40.521329 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-19647e-6849d859cd-txmcp"] Sep 11 19:14:40.577281 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:40.577244 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:41.172117 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:41.172083 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e5dca55-4746-454a-8746-2c54d3f1fa60" path="/var/lib/kubelet/pods/7e5dca55-4746-454a-8746-2c54d3f1fa60/volumes" Sep 11 19:14:41.172650 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:41.172624 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d67ab2c-ee3a-448c-8374-16e1bc25e089" path="/var/lib/kubelet/pods/9d67ab2c-ee3a-448c-8374-16e1bc25e089/volumes" Sep 11 19:14:41.173077 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:41.173061 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce37bbd2-eb10-4c58-91c5-fdf15fa56810" path="/var/lib/kubelet/pods/ce37bbd2-eb10-4c58-91c5-fdf15fa56810/volumes" Sep 11 19:14:41.342448 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:41.342408 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:42.063587 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:42.063533 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:44.570481 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:44.570445 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:44.666378 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:44.666344 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:45.092003 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:45.091960 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:45.757409 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:45.757371 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:45.858003 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:45.857971 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:46.005259 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:46.005230 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:46.117366 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:46.117326 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:46.576759 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:46.576722 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:47.459116 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:47.459079 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:48.280282 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:48.280250 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:49.065395 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:49.065361 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:49.911350 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:49.911314 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:50.857101 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:50.857062 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:53.004235 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:53.004197 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:56.874209 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:56.874180 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn"] Sep 11 19:14:56.874664 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:56.874365 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" podUID="70526a66-5d7f-4170-a649-2548ddbaac4a" containerName="maas-api" containerID="cri-o://eeeb838b01b991459f78a5c943b7baedcdbb9dc6059ac73858ffb985797827c0" gracePeriod=30 Sep 11 19:14:56.885212 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:56.885186 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:56.923898 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:56.923863 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj"] Sep 11 19:14:56.924090 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:56.924067 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" podUID="a37e489f-3485-47cb-9c43-67f0efdfb8e5" containerName="payload-pre-processing" containerID="cri-o://39ccf22d214c7e29c81470369358cab71bd5ce59a3a7784aed27aefc229f9831" gracePeriod=30 Sep 11 19:14:56.986314 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:56.986272 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:57.081777 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.081744 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:57.142459 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.142373 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:57.184966 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.184935 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:14:57.203179 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.203155 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" Sep 11 19:14:57.297261 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.297230 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70526a66-5d7f-4170-a649-2548ddbaac4a-maas-api-tls\") pod \"70526a66-5d7f-4170-a649-2548ddbaac4a\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " Sep 11 19:14:57.297443 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.297274 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22cvb\" (UniqueName: \"kubernetes.io/projected/70526a66-5d7f-4170-a649-2548ddbaac4a-kube-api-access-22cvb\") pod \"70526a66-5d7f-4170-a649-2548ddbaac4a\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " Sep 11 19:14:57.297443 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.297307 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37e489f-3485-47cb-9c43-67f0efdfb8e5-plugins-config-volume\") pod \"a37e489f-3485-47cb-9c43-67f0efdfb8e5\" (UID: \"a37e489f-3485-47cb-9c43-67f0efdfb8e5\") " Sep 11 19:14:57.297443 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.297377 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70526a66-5d7f-4170-a649-2548ddbaac4a-openshift-service-ca\") pod \"70526a66-5d7f-4170-a649-2548ddbaac4a\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " Sep 11 19:14:57.297443 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.297412 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70526a66-5d7f-4170-a649-2548ddbaac4a-metrics-certs\") pod \"70526a66-5d7f-4170-a649-2548ddbaac4a\" (UID: \"70526a66-5d7f-4170-a649-2548ddbaac4a\") " Sep 11 19:14:57.297443 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.297440 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtm4s\" (UniqueName: \"kubernetes.io/projected/a37e489f-3485-47cb-9c43-67f0efdfb8e5-kube-api-access-xtm4s\") pod \"a37e489f-3485-47cb-9c43-67f0efdfb8e5\" (UID: \"a37e489f-3485-47cb-9c43-67f0efdfb8e5\") " Sep 11 19:14:57.297809 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.297782 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37e489f-3485-47cb-9c43-67f0efdfb8e5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a37e489f-3485-47cb-9c43-67f0efdfb8e5" (UID: "a37e489f-3485-47cb-9c43-67f0efdfb8e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:14:57.297890 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.297809 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70526a66-5d7f-4170-a649-2548ddbaac4a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "70526a66-5d7f-4170-a649-2548ddbaac4a" (UID: "70526a66-5d7f-4170-a649-2548ddbaac4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:14:57.299469 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.299447 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70526a66-5d7f-4170-a649-2548ddbaac4a-kube-api-access-22cvb" (OuterVolumeSpecName: "kube-api-access-22cvb") pod "70526a66-5d7f-4170-a649-2548ddbaac4a" (UID: "70526a66-5d7f-4170-a649-2548ddbaac4a"). InnerVolumeSpecName "kube-api-access-22cvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:14:57.299597 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.299524 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37e489f-3485-47cb-9c43-67f0efdfb8e5-kube-api-access-xtm4s" (OuterVolumeSpecName: "kube-api-access-xtm4s") pod "a37e489f-3485-47cb-9c43-67f0efdfb8e5" (UID: "a37e489f-3485-47cb-9c43-67f0efdfb8e5"). InnerVolumeSpecName "kube-api-access-xtm4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:14:57.299846 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.299811 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70526a66-5d7f-4170-a649-2548ddbaac4a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "70526a66-5d7f-4170-a649-2548ddbaac4a" (UID: "70526a66-5d7f-4170-a649-2548ddbaac4a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:14:57.299846 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.299824 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70526a66-5d7f-4170-a649-2548ddbaac4a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "70526a66-5d7f-4170-a649-2548ddbaac4a" (UID: "70526a66-5d7f-4170-a649-2548ddbaac4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:14:57.398844 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.398766 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70526a66-5d7f-4170-a649-2548ddbaac4a-openshift-service-ca\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:57.398844 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.398795 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70526a66-5d7f-4170-a649-2548ddbaac4a-metrics-certs\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:57.398844 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.398803 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtm4s\" (UniqueName: \"kubernetes.io/projected/a37e489f-3485-47cb-9c43-67f0efdfb8e5-kube-api-access-xtm4s\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:57.398844 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.398811 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70526a66-5d7f-4170-a649-2548ddbaac4a-maas-api-tls\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:57.398844 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.398819 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22cvb\" (UniqueName: \"kubernetes.io/projected/70526a66-5d7f-4170-a649-2548ddbaac4a-kube-api-access-22cvb\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:57.398844 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.398827 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37e489f-3485-47cb-9c43-67f0efdfb8e5-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:14:57.487661 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.487628 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:57.572812 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.572781 2556 generic.go:356] "Generic (PLEG): container finished" podID="70526a66-5d7f-4170-a649-2548ddbaac4a" containerID="eeeb838b01b991459f78a5c943b7baedcdbb9dc6059ac73858ffb985797827c0" exitCode=0 Sep 11 19:14:57.572994 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.572831 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" Sep 11 19:14:57.572994 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.572870 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" event={"ID":"70526a66-5d7f-4170-a649-2548ddbaac4a","Type":"ContainerDied","Data":"eeeb838b01b991459f78a5c943b7baedcdbb9dc6059ac73858ffb985797827c0"} Sep 11 19:14:57.572994 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.572921 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn" event={"ID":"70526a66-5d7f-4170-a649-2548ddbaac4a","Type":"ContainerDied","Data":"1decc335a1e9cdcc9033ca1001060cdfcd8fd4a1a0b905db7382044d8da645c4"} Sep 11 19:14:57.572994 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.572943 2556 scope.go:117] "RemoveContainer" containerID="eeeb838b01b991459f78a5c943b7baedcdbb9dc6059ac73858ffb985797827c0" Sep 11 19:14:57.574605 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.574575 2556 generic.go:356] "Generic (PLEG): container finished" podID="a37e489f-3485-47cb-9c43-67f0efdfb8e5" containerID="39ccf22d214c7e29c81470369358cab71bd5ce59a3a7784aed27aefc229f9831" exitCode=0 Sep 11 19:14:57.574734 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.574622 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" Sep 11 19:14:57.574734 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.574664 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" event={"ID":"a37e489f-3485-47cb-9c43-67f0efdfb8e5","Type":"ContainerDied","Data":"39ccf22d214c7e29c81470369358cab71bd5ce59a3a7784aed27aefc229f9831"} Sep 11 19:14:57.574734 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.574721 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj" event={"ID":"a37e489f-3485-47cb-9c43-67f0efdfb8e5","Type":"ContainerDied","Data":"52f2a96650f48bd318d3063c467d9f14b2946e23082fd004a667a40e902c950b"} Sep 11 19:14:57.582765 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.582658 2556 scope.go:117] "RemoveContainer" containerID="eeeb838b01b991459f78a5c943b7baedcdbb9dc6059ac73858ffb985797827c0" Sep 11 19:14:57.583021 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:14:57.582996 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeeb838b01b991459f78a5c943b7baedcdbb9dc6059ac73858ffb985797827c0\": container with ID starting with eeeb838b01b991459f78a5c943b7baedcdbb9dc6059ac73858ffb985797827c0 not found: ID does not exist" containerID="eeeb838b01b991459f78a5c943b7baedcdbb9dc6059ac73858ffb985797827c0" Sep 11 19:14:57.583083 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.583030 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeeb838b01b991459f78a5c943b7baedcdbb9dc6059ac73858ffb985797827c0"} err="failed to get container status \"eeeb838b01b991459f78a5c943b7baedcdbb9dc6059ac73858ffb985797827c0\": rpc error: code = NotFound desc = could not find container \"eeeb838b01b991459f78a5c943b7baedcdbb9dc6059ac73858ffb985797827c0\": container with ID starting with eeeb838b01b991459f78a5c943b7baedcdbb9dc6059ac73858ffb985797827c0 not found: ID does not exist" Sep 11 19:14:57.583083 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.583048 2556 scope.go:117] "RemoveContainer" containerID="39ccf22d214c7e29c81470369358cab71bd5ce59a3a7784aed27aefc229f9831" Sep 11 19:14:57.587625 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.587606 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:57.591314 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.591297 2556 scope.go:117] "RemoveContainer" containerID="39ccf22d214c7e29c81470369358cab71bd5ce59a3a7784aed27aefc229f9831" Sep 11 19:14:57.591573 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:14:57.591551 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ccf22d214c7e29c81470369358cab71bd5ce59a3a7784aed27aefc229f9831\": container with ID starting with 39ccf22d214c7e29c81470369358cab71bd5ce59a3a7784aed27aefc229f9831 not found: ID does not exist" containerID="39ccf22d214c7e29c81470369358cab71bd5ce59a3a7784aed27aefc229f9831" Sep 11 19:14:57.591634 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.591580 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ccf22d214c7e29c81470369358cab71bd5ce59a3a7784aed27aefc229f9831"} err="failed to get container status \"39ccf22d214c7e29c81470369358cab71bd5ce59a3a7784aed27aefc229f9831\": rpc error: code = NotFound desc = could not find container \"39ccf22d214c7e29c81470369358cab71bd5ce59a3a7784aed27aefc229f9831\": container with ID starting with 39ccf22d214c7e29c81470369358cab71bd5ce59a3a7784aed27aefc229f9831 not found: ID does not exist" Sep 11 19:14:57.598737 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.598717 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj"] Sep 11 19:14:57.602478 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.602459 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-37ce96-595f449bfd-jmvlj"] Sep 11 19:14:57.612029 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.612009 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn"] Sep 11 19:14:57.615794 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:57.615775 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37ce96-6576fff95b-w28mn"] Sep 11 19:14:58.270082 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:58.269997 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:59.061937 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:59.061901 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:14:59.171729 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:59.171696 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70526a66-5d7f-4170-a649-2548ddbaac4a" path="/var/lib/kubelet/pods/70526a66-5d7f-4170-a649-2548ddbaac4a/volumes" Sep 11 19:14:59.172046 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:14:59.172032 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a37e489f-3485-47cb-9c43-67f0efdfb8e5" path="/var/lib/kubelet/pods/a37e489f-3485-47cb-9c43-67f0efdfb8e5/volumes" Sep 11 19:15:00.131462 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.131427 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8280eb-29819235-j6fv7"] Sep 11 19:15:00.131887 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.131871 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55" containerName="payload-processing" Sep 11 19:15:00.131935 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.131889 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55" containerName="payload-processing" Sep 11 19:15:00.131935 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.131902 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a333da-fe91-4593-b69e-f49b76efba30" containerName="payload-pre-processing" Sep 11 19:15:00.131935 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.131908 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a333da-fe91-4593-b69e-f49b76efba30" containerName="payload-pre-processing" Sep 11 19:15:00.131935 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.131919 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5dca55-4746-454a-8746-2c54d3f1fa60" containerName="payload-processing" Sep 11 19:15:00.131935 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.131928 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5dca55-4746-454a-8746-2c54d3f1fa60" containerName="payload-processing" Sep 11 19:15:00.132085 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.131937 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37e489f-3485-47cb-9c43-67f0efdfb8e5" containerName="payload-pre-processing" Sep 11 19:15:00.132085 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.131943 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37e489f-3485-47cb-9c43-67f0efdfb8e5" containerName="payload-pre-processing" Sep 11 19:15:00.132085 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.131962 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d67ab2c-ee3a-448c-8374-16e1bc25e089" containerName="payload-pre-processing" Sep 11 19:15:00.132085 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.131967 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d67ab2c-ee3a-448c-8374-16e1bc25e089" containerName="payload-pre-processing" Sep 11 19:15:00.132085 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.131974 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70526a66-5d7f-4170-a649-2548ddbaac4a" containerName="maas-api" Sep 11 19:15:00.132085 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.131979 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="70526a66-5d7f-4170-a649-2548ddbaac4a" containerName="maas-api" Sep 11 19:15:00.132085 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.131985 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85d664d0-b623-4325-ae66-f577a10db529" containerName="payload-processing" Sep 11 19:15:00.132085 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.131990 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="85d664d0-b623-4325-ae66-f577a10db529" containerName="payload-processing" Sep 11 19:15:00.132085 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.132001 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce37bbd2-eb10-4c58-91c5-fdf15fa56810" containerName="maas-api" Sep 11 19:15:00.132085 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.132009 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce37bbd2-eb10-4c58-91c5-fdf15fa56810" containerName="maas-api" Sep 11 19:15:00.132085 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.132065 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="70526a66-5d7f-4170-a649-2548ddbaac4a" containerName="maas-api" Sep 11 19:15:00.132085 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.132071 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce37bbd2-eb10-4c58-91c5-fdf15fa56810" containerName="maas-api" Sep 11 19:15:00.132085 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.132081 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5dca55-4746-454a-8746-2c54d3f1fa60" containerName="payload-processing" Sep 11 19:15:00.132460 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.132091 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b9e6360-c6e0-469d-bbbb-e8dd0cf7fe55" containerName="payload-processing" Sep 11 19:15:00.132460 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.132097 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="85d664d0-b623-4325-ae66-f577a10db529" containerName="payload-processing" Sep 11 19:15:00.132460 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.132105 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a333da-fe91-4593-b69e-f49b76efba30" containerName="payload-pre-processing" Sep 11 19:15:00.132460 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.132113 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d67ab2c-ee3a-448c-8374-16e1bc25e089" containerName="payload-pre-processing" Sep 11 19:15:00.132460 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.132119 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a37e489f-3485-47cb-9c43-67f0efdfb8e5" containerName="payload-pre-processing" Sep 11 19:15:00.136921 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.136904 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8280eb-29819235-j6fv7" Sep 11 19:15:00.138964 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.138944 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\"" Sep 11 19:15:00.143525 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.143501 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8280eb-29819235-j6fv7"] Sep 11 19:15:00.149062 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.149043 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8280eb-29819235-j6fv7" Sep 11 19:15:00.427039 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.426968 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8280eb-29819235-j6fv7"] Sep 11 19:15:00.429633 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:15:00.429607 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c95bf5_8d14_418c_b236_792e71442b33.slice/crio-92e67bf2a297c89245eb9d1d6c63e640d12102cb4f131a796f8c192605eee5b5 WatchSource:0}: Error finding container 92e67bf2a297c89245eb9d1d6c63e640d12102cb4f131a796f8c192605eee5b5: Status 404 returned error can't find the container with id 92e67bf2a297c89245eb9d1d6c63e640d12102cb4f131a796f8c192605eee5b5 Sep 11 19:15:00.594130 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:00.594096 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8280eb-29819235-j6fv7" event={"ID":"b7c95bf5-8d14-418c-b236-792e71442b33","Type":"ContainerStarted","Data":"92e67bf2a297c89245eb9d1d6c63e640d12102cb4f131a796f8c192605eee5b5"} Sep 11 19:15:01.601340 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:01.601308 2556 generic.go:356] "Generic (PLEG): container finished" podID="b7c95bf5-8d14-418c-b236-792e71442b33" containerID="c8e6f93a8633e5e3d0fab39a94c014c488017accbb43e3a3edf36b20b2f53012" exitCode=0 Sep 11 19:15:01.601823 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:01.601395 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8280eb-29819235-j6fv7" event={"ID":"b7c95bf5-8d14-418c-b236-792e71442b33","Type":"ContainerDied","Data":"c8e6f93a8633e5e3d0fab39a94c014c488017accbb43e3a3edf36b20b2f53012"} Sep 11 19:15:02.739393 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:02.739371 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8280eb-29819235-j6fv7" Sep 11 19:15:03.294196 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:03.294163 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:15:03.354502 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:03.354468 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:15:03.612822 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:03.612789 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8280eb-29819235-j6fv7" Sep 11 19:15:03.612822 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:03.612797 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8280eb-29819235-j6fv7" event={"ID":"b7c95bf5-8d14-418c-b236-792e71442b33","Type":"ContainerDied","Data":"92e67bf2a297c89245eb9d1d6c63e640d12102cb4f131a796f8c192605eee5b5"} Sep 11 19:15:03.613039 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:03.612851 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92e67bf2a297c89245eb9d1d6c63e640d12102cb4f131a796f8c192605eee5b5" Sep 11 19:15:03.772768 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:03.772737 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:15:15.060837 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:15:15.060735 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:16:06.840984 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:16:06.840954 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:16:08.460976 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:16:08.460936 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:16:11.664782 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:16:11.664738 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:16:52.040907 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:16:52.040866 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:17:11.265323 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:17:11.265285 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:17:43.967412 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:17:43.967376 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:07.617484 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:07.617392 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:08.458064 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:08.458023 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:08.876863 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:08.876827 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:09.173282 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:09.173196 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:09.333104 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:09.333068 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:11.972702 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:11.972659 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:12.070291 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:12.070265 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:12.952081 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:12.952039 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:13.765181 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:13.765141 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:13.869625 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:13.869586 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:13.997392 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:13.997351 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:14.358119 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.358080 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:14.454449 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.454421 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:14.489567 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.489525 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz"] Sep 11 19:18:14.489966 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.489950 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c95bf5-8d14-418c-b236-792e71442b33" containerName="cleanup" Sep 11 19:18:14.489966 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.489967 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c95bf5-8d14-418c-b236-792e71442b33" containerName="cleanup" Sep 11 19:18:14.490087 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.490055 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7c95bf5-8d14-418c-b236-792e71442b33" containerName="cleanup" Sep 11 19:18:14.493255 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.493236 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" Sep 11 19:18:14.495699 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.495678 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-021613\"" Sep 11 19:18:14.495803 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.495699 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-021613-dockercfg-cmnrt\"" Sep 11 19:18:14.503248 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.503222 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz"] Sep 11 19:18:14.590002 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.589968 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc74l\" (UniqueName: \"kubernetes.io/projected/8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1-kube-api-access-lc74l\") pod \"payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz\" (UID: \"8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" Sep 11 19:18:14.590199 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.590022 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz\" (UID: \"8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" Sep 11 19:18:14.691052 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.690969 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc74l\" (UniqueName: \"kubernetes.io/projected/8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1-kube-api-access-lc74l\") pod \"payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz\" (UID: \"8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" Sep 11 19:18:14.691052 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.691016 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz\" (UID: \"8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" Sep 11 19:18:14.691712 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.691693 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz\" (UID: \"8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" Sep 11 19:18:14.700187 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.700159 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc74l\" (UniqueName: \"kubernetes.io/projected/8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1-kube-api-access-lc74l\") pod \"payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz\" (UID: \"8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" Sep 11 19:18:14.764841 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.764792 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:14.805644 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:14.805618 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" Sep 11 19:18:15.078348 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:15.078266 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz"] Sep 11 19:18:15.080657 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:18:15.080631 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb5baaa_6c60_4158_bc86_40fd1a6b0cd1.slice/crio-e0cbc91115ee77a60d93d26d0852566a63a3e377ae0547aef9276f0a05ed370f WatchSource:0}: Error finding container e0cbc91115ee77a60d93d26d0852566a63a3e377ae0547aef9276f0a05ed370f: Status 404 returned error can't find the container with id e0cbc91115ee77a60d93d26d0852566a63a3e377ae0547aef9276f0a05ed370f Sep 11 19:18:15.158502 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:15.158476 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 19:18:15.164967 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:15.164949 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 19:18:15.553831 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:15.553792 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:15.586349 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:15.586315 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" event={"ID":"8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1","Type":"ContainerStarted","Data":"4e5a06da80b88c0e5e9f29551bbdcb4b37a43912f113e0e4479d4b321d406c3c"} Sep 11 19:18:15.586493 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:15.586372 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" event={"ID":"8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1","Type":"ContainerStarted","Data":"e0cbc91115ee77a60d93d26d0852566a63a3e377ae0547aef9276f0a05ed370f"} Sep 11 19:18:15.586493 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:15.586421 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" Sep 11 19:18:15.602525 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:15.602474 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" podStartSLOduration=1.602455885 podStartE2EDuration="1.602455885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:18:15.601182499 +0000 UTC m=+1801.054988473" watchObservedRunningTime="2026-09-11 19:18:15.602455885 +0000 UTC m=+1801.056261841" Sep 11 19:18:26.594118 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:26.594084 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" Sep 11 19:18:36.019779 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:36.019743 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:36.735784 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:36.735745 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:36.971215 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:36.971178 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:37.575499 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:37.575464 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:37.676507 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:37.676468 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:37.927258 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:37.927137 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:38.095284 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:38.095242 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:38.366500 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:38.366446 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:41.075519 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:41.075483 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:41.174877 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:41.174842 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:41.274940 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:41.274849 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:41.954164 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:41.954132 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:42.765824 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:42.765788 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:43.593503 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:43.593476 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:43.687587 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:43.687514 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:43.941681 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:43.941586 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:44.079329 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:44.079292 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:44.183800 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:44.183765 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:18:44.226423 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:44.226326 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm"] Sep 11 19:18:44.229910 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:44.229892 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" Sep 11 19:18:44.232141 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:44.232118 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-afd85c\"" Sep 11 19:18:44.232141 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:44.232130 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-afd85c-dockercfg-hzhrt\"" Sep 11 19:18:44.239016 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:44.238992 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm"] Sep 11 19:18:44.275408 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:44.275370 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfkxq\" (UniqueName: \"kubernetes.io/projected/493c5ad8-37e6-4e1a-a223-93eeb69c37d9-kube-api-access-vfkxq\") pod \"payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm\" (UID: \"493c5ad8-37e6-4e1a-a223-93eeb69c37d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" Sep 11 19:18:44.275621 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:44.275493 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/493c5ad8-37e6-4e1a-a223-93eeb69c37d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm\" (UID: \"493c5ad8-37e6-4e1a-a223-93eeb69c37d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" Sep 11 19:18:44.376442 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:44.376395 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfkxq\" (UniqueName: \"kubernetes.io/projected/493c5ad8-37e6-4e1a-a223-93eeb69c37d9-kube-api-access-vfkxq\") pod \"payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm\" (UID: \"493c5ad8-37e6-4e1a-a223-93eeb69c37d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" Sep 11 19:18:44.376643 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:44.376483 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/493c5ad8-37e6-4e1a-a223-93eeb69c37d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm\" (UID: \"493c5ad8-37e6-4e1a-a223-93eeb69c37d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" Sep 11 19:18:44.377257 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:44.377236 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/493c5ad8-37e6-4e1a-a223-93eeb69c37d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm\" (UID: \"493c5ad8-37e6-4e1a-a223-93eeb69c37d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" Sep 11 19:18:44.384121 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:44.384094 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfkxq\" (UniqueName: \"kubernetes.io/projected/493c5ad8-37e6-4e1a-a223-93eeb69c37d9-kube-api-access-vfkxq\") pod \"payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm\" (UID: \"493c5ad8-37e6-4e1a-a223-93eeb69c37d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" Sep 11 19:18:44.542378 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:44.542339 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" Sep 11 19:18:44.813238 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:44.813214 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm"] Sep 11 19:18:44.816035 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:18:44.816006 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493c5ad8_37e6_4e1a_a223_93eeb69c37d9.slice/crio-a0ef3a56f3ee2f72378b0b8d243b7e0dcfa0d569973dcabbd4bb95e5e6ebac03 WatchSource:0}: Error finding container a0ef3a56f3ee2f72378b0b8d243b7e0dcfa0d569973dcabbd4bb95e5e6ebac03: Status 404 returned error can't find the container with id a0ef3a56f3ee2f72378b0b8d243b7e0dcfa0d569973dcabbd4bb95e5e6ebac03 Sep 11 19:18:45.738997 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:45.738956 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" event={"ID":"493c5ad8-37e6-4e1a-a223-93eeb69c37d9","Type":"ContainerStarted","Data":"4662d2feb26c8e3352a31413ceb8620b88a7f4c3c04fd41f361d5101cb2361bb"} Sep 11 19:18:45.739389 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:45.739019 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" event={"ID":"493c5ad8-37e6-4e1a-a223-93eeb69c37d9","Type":"ContainerStarted","Data":"a0ef3a56f3ee2f72378b0b8d243b7e0dcfa0d569973dcabbd4bb95e5e6ebac03"} Sep 11 19:18:45.739389 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:45.739044 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" Sep 11 19:18:45.754880 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:45.754824 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" podStartSLOduration=1.754808817 podStartE2EDuration="1.754808817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:18:45.752962225 +0000 UTC m=+1831.206768195" watchObservedRunningTime="2026-09-11 19:18:45.754808817 +0000 UTC m=+1831.208614771" Sep 11 19:18:56.746516 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:56.746486 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" Sep 11 19:18:57.852607 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:18:57.852567 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:05.567029 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:05.566994 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:06.096074 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:06.096043 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:06.473754 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:06.473645 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:07.199350 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:07.199312 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:08.058289 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:08.058248 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:08.743930 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:08.743894 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:11.174550 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:11.174505 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm_493c5ad8-37e6-4e1a-a223-93eeb69c37d9/payload-processing/0.log" Sep 11 19:19:24.082703 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:24.082658 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:46.958029 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:46.957998 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz_8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1/payload-processing/0.log" Sep 11 19:19:47.218881 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:47.218808 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm_493c5ad8-37e6-4e1a-a223-93eeb69c37d9/payload-processing/0.log" Sep 11 19:19:47.483903 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:47.483824 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:47.804235 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:47.804199 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:48.381616 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:48.381587 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:49.909974 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:49.909943 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:50.722434 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:50.722400 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:50.817216 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:50.817183 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:50.966664 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:50.966631 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:51.857815 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:51.857782 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:52.778518 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:52.778482 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:53.526510 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:53.526476 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:54.343488 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.343455 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv"] Sep 11 19:19:54.347717 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.347692 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:19:54.349914 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.349895 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-75b1ed-serving-cert\"" Sep 11 19:19:54.357919 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.356909 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv"] Sep 11 19:19:54.389977 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.389931 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:54.490455 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.490420 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:54.534398 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.534359 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24509d56-8ec0-4f45-995a-846cd599146b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv\" (UID: \"24509d56-8ec0-4f45-995a-846cd599146b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:19:54.534612 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.534433 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24509d56-8ec0-4f45-995a-846cd599146b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv\" (UID: \"24509d56-8ec0-4f45-995a-846cd599146b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:19:54.534612 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.534487 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hmnd\" (UniqueName: \"kubernetes.io/projected/24509d56-8ec0-4f45-995a-846cd599146b-kube-api-access-2hmnd\") pod \"maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv\" (UID: \"24509d56-8ec0-4f45-995a-846cd599146b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:19:54.534612 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.534562 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24509d56-8ec0-4f45-995a-846cd599146b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv\" (UID: \"24509d56-8ec0-4f45-995a-846cd599146b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:19:54.619442 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.619353 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:54.635550 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.635521 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24509d56-8ec0-4f45-995a-846cd599146b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv\" (UID: \"24509d56-8ec0-4f45-995a-846cd599146b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:19:54.635637 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.635602 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hmnd\" (UniqueName: \"kubernetes.io/projected/24509d56-8ec0-4f45-995a-846cd599146b-kube-api-access-2hmnd\") pod \"maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv\" (UID: \"24509d56-8ec0-4f45-995a-846cd599146b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:19:54.635689 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.635648 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24509d56-8ec0-4f45-995a-846cd599146b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv\" (UID: \"24509d56-8ec0-4f45-995a-846cd599146b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:19:54.635751 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.635688 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24509d56-8ec0-4f45-995a-846cd599146b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv\" (UID: \"24509d56-8ec0-4f45-995a-846cd599146b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:19:54.636288 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.636259 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24509d56-8ec0-4f45-995a-846cd599146b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv\" (UID: \"24509d56-8ec0-4f45-995a-846cd599146b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:19:54.637994 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.637973 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24509d56-8ec0-4f45-995a-846cd599146b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv\" (UID: \"24509d56-8ec0-4f45-995a-846cd599146b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:19:54.638095 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.638049 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24509d56-8ec0-4f45-995a-846cd599146b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv\" (UID: \"24509d56-8ec0-4f45-995a-846cd599146b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:19:54.642994 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.642974 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hmnd\" (UniqueName: \"kubernetes.io/projected/24509d56-8ec0-4f45-995a-846cd599146b-kube-api-access-2hmnd\") pod \"maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv\" (UID: \"24509d56-8ec0-4f45-995a-846cd599146b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:19:54.660813 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.660787 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:19:54.934198 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.933967 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv"] Sep 11 19:19:54.936681 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:19:54.936655 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24509d56_8ec0_4f45_995a_846cd599146b.slice/crio-8284924c788cf894ace274d802ef8492bd5aa48b023ae49302c01ba0990f146a WatchSource:0}: Error finding container 8284924c788cf894ace274d802ef8492bd5aa48b023ae49302c01ba0990f146a: Status 404 returned error can't find the container with id 8284924c788cf894ace274d802ef8492bd5aa48b023ae49302c01ba0990f146a Sep 11 19:19:54.938443 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.938423 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 19:19:54.977930 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:54.977897 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:55.077658 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:55.077592 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:19:55.090741 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:55.090701 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" event={"ID":"24509d56-8ec0-4f45-995a-846cd599146b","Type":"ContainerStarted","Data":"8284924c788cf894ace274d802ef8492bd5aa48b023ae49302c01ba0990f146a"} Sep 11 19:19:56.097122 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:56.097090 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" event={"ID":"24509d56-8ec0-4f45-995a-846cd599146b","Type":"ContainerStarted","Data":"8fe9123fbceef1e35e027142c48d6ea44a393a86d3aa7e9e267f35552fcb79dd"} Sep 11 19:19:56.097567 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:56.097213 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:19:56.116523 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:19:56.116473 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" podStartSLOduration=1.6335406209999999 podStartE2EDuration="2.116457466s" totalImagesPullingTime="482.916845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:19:56.114730168 +0000 UTC m=+1901.568536142" watchObservedRunningTime="2026-09-11 19:19:56.116457466 +0000 UTC m=+1901.570263479" Sep 11 19:20:02.107872 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:02.107843 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:20:16.673622 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:16.673585 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:17.410702 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:17.410659 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:18.250991 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:18.250938 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:18.989959 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:18.989916 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:19.734140 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:19.734105 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:20.647226 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:20.647186 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:20.746403 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:20.746355 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:22.299901 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:22.299865 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:23.173448 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:23.173402 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:23.414918 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:23.414883 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:24.100750 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:24.100715 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:24.861471 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:24.861444 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:25.669429 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:25.669398 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:26.488590 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:26.488549 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:32.517705 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.517673 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv"] Sep 11 19:20:32.518199 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.517942 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" podUID="24509d56-8ec0-4f45-995a-846cd599146b" containerName="maas-api" containerID="cri-o://8fe9123fbceef1e35e027142c48d6ea44a393a86d3aa7e9e267f35552fcb79dd" gracePeriod=30 Sep 11 19:20:32.546751 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.546694 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:32.620767 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.620736 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:32.721058 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.721015 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:32.766238 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.766216 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:20:32.799645 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.799607 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:32.894179 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.894146 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24509d56-8ec0-4f45-995a-846cd599146b-metrics-certs\") pod \"24509d56-8ec0-4f45-995a-846cd599146b\" (UID: \"24509d56-8ec0-4f45-995a-846cd599146b\") " Sep 11 19:20:32.894360 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.894209 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24509d56-8ec0-4f45-995a-846cd599146b-maas-api-tls\") pod \"24509d56-8ec0-4f45-995a-846cd599146b\" (UID: \"24509d56-8ec0-4f45-995a-846cd599146b\") " Sep 11 19:20:32.894360 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.894269 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24509d56-8ec0-4f45-995a-846cd599146b-openshift-service-ca\") pod \"24509d56-8ec0-4f45-995a-846cd599146b\" (UID: \"24509d56-8ec0-4f45-995a-846cd599146b\") " Sep 11 19:20:32.894489 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.894362 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hmnd\" (UniqueName: \"kubernetes.io/projected/24509d56-8ec0-4f45-995a-846cd599146b-kube-api-access-2hmnd\") pod \"24509d56-8ec0-4f45-995a-846cd599146b\" (UID: \"24509d56-8ec0-4f45-995a-846cd599146b\") " Sep 11 19:20:32.894810 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.894779 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24509d56-8ec0-4f45-995a-846cd599146b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24509d56-8ec0-4f45-995a-846cd599146b" (UID: "24509d56-8ec0-4f45-995a-846cd599146b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:20:32.896612 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.896583 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24509d56-8ec0-4f45-995a-846cd599146b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "24509d56-8ec0-4f45-995a-846cd599146b" (UID: "24509d56-8ec0-4f45-995a-846cd599146b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:20:32.896869 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.896840 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24509d56-8ec0-4f45-995a-846cd599146b-kube-api-access-2hmnd" (OuterVolumeSpecName: "kube-api-access-2hmnd") pod "24509d56-8ec0-4f45-995a-846cd599146b" (UID: "24509d56-8ec0-4f45-995a-846cd599146b"). InnerVolumeSpecName "kube-api-access-2hmnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:20:32.896943 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.896918 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24509d56-8ec0-4f45-995a-846cd599146b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24509d56-8ec0-4f45-995a-846cd599146b" (UID: "24509d56-8ec0-4f45-995a-846cd599146b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:20:32.995185 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.995148 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hmnd\" (UniqueName: \"kubernetes.io/projected/24509d56-8ec0-4f45-995a-846cd599146b-kube-api-access-2hmnd\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:20:32.995185 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.995181 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24509d56-8ec0-4f45-995a-846cd599146b-metrics-certs\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:20:32.995185 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.995190 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24509d56-8ec0-4f45-995a-846cd599146b-maas-api-tls\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:20:32.995417 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:32.995198 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24509d56-8ec0-4f45-995a-846cd599146b-openshift-service-ca\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:20:33.129172 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:33.129069 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:33.229325 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:33.229295 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:33.282584 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:33.282526 2556 generic.go:356] "Generic (PLEG): container finished" podID="24509d56-8ec0-4f45-995a-846cd599146b" containerID="8fe9123fbceef1e35e027142c48d6ea44a393a86d3aa7e9e267f35552fcb79dd" exitCode=0 Sep 11 19:20:33.282746 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:33.282591 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" Sep 11 19:20:33.282746 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:33.282615 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" event={"ID":"24509d56-8ec0-4f45-995a-846cd599146b","Type":"ContainerDied","Data":"8fe9123fbceef1e35e027142c48d6ea44a393a86d3aa7e9e267f35552fcb79dd"} Sep 11 19:20:33.282746 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:33.282666 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv" event={"ID":"24509d56-8ec0-4f45-995a-846cd599146b","Type":"ContainerDied","Data":"8284924c788cf894ace274d802ef8492bd5aa48b023ae49302c01ba0990f146a"} Sep 11 19:20:33.282746 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:33.282691 2556 scope.go:117] "RemoveContainer" containerID="8fe9123fbceef1e35e027142c48d6ea44a393a86d3aa7e9e267f35552fcb79dd" Sep 11 19:20:33.291466 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:33.291452 2556 scope.go:117] "RemoveContainer" containerID="8fe9123fbceef1e35e027142c48d6ea44a393a86d3aa7e9e267f35552fcb79dd" Sep 11 19:20:33.291768 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:20:33.291749 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe9123fbceef1e35e027142c48d6ea44a393a86d3aa7e9e267f35552fcb79dd\": container with ID starting with 8fe9123fbceef1e35e027142c48d6ea44a393a86d3aa7e9e267f35552fcb79dd not found: ID does not exist" containerID="8fe9123fbceef1e35e027142c48d6ea44a393a86d3aa7e9e267f35552fcb79dd" Sep 11 19:20:33.291847 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:33.291780 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe9123fbceef1e35e027142c48d6ea44a393a86d3aa7e9e267f35552fcb79dd"} err="failed to get container status \"8fe9123fbceef1e35e027142c48d6ea44a393a86d3aa7e9e267f35552fcb79dd\": rpc error: code = NotFound desc = could not find container \"8fe9123fbceef1e35e027142c48d6ea44a393a86d3aa7e9e267f35552fcb79dd\": container with ID starting with 8fe9123fbceef1e35e027142c48d6ea44a393a86d3aa7e9e267f35552fcb79dd not found: ID does not exist" Sep 11 19:20:33.299944 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:33.299918 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv"] Sep 11 19:20:33.302651 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:33.302633 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75b1ed-57d7745d85-ncpwv"] Sep 11 19:20:34.074048 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:34.073994 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:34.758935 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:34.758890 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:35.172398 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:35.172361 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24509d56-8ec0-4f45-995a-846cd599146b" path="/var/lib/kubelet/pods/24509d56-8ec0-4f45-995a-846cd599146b/volumes" Sep 11 19:20:35.456113 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:35.455996 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:36.263145 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:36.263111 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:38.781768 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:38.781736 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:38.882702 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:38.882665 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:39.049137 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:39.049103 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:43.248652 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:43.248622 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:44.076625 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:44.076594 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:44.775791 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:44.775757 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:45.472214 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:45.472179 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:46.258993 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:46.258957 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:58.162446 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.162410 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:58.190858 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.190827 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz"] Sep 11 19:20:58.191119 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.191092 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" podUID="8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1" containerName="payload-processing" containerID="cri-o://4e5a06da80b88c0e5e9f29551bbdcb4b37a43912f113e0e4479d4b321d406c3c" gracePeriod=30 Sep 11 19:20:58.261000 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.260956 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:58.360009 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.359972 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:58.409746 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.409718 2556 generic.go:356] "Generic (PLEG): container finished" podID="8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1" containerID="4e5a06da80b88c0e5e9f29551bbdcb4b37a43912f113e0e4479d4b321d406c3c" exitCode=0 Sep 11 19:20:58.409901 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.409785 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" event={"ID":"8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1","Type":"ContainerDied","Data":"4e5a06da80b88c0e5e9f29551bbdcb4b37a43912f113e0e4479d4b321d406c3c"} Sep 11 19:20:58.427168 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.427087 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:58.456968 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.456945 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" Sep 11 19:20:58.534070 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.533936 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1-plugins-config-volume\") pod \"8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1\" (UID: \"8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1\") " Sep 11 19:20:58.534264 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.534114 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc74l\" (UniqueName: \"kubernetes.io/projected/8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1-kube-api-access-lc74l\") pod \"8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1\" (UID: \"8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1\") " Sep 11 19:20:58.534917 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.534879 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1" (UID: "8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:20:58.543392 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.543349 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1-kube-api-access-lc74l" (OuterVolumeSpecName: "kube-api-access-lc74l") pod "8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1" (UID: "8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1"). InnerVolumeSpecName "kube-api-access-lc74l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:20:58.635450 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.635415 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc74l\" (UniqueName: \"kubernetes.io/projected/8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1-kube-api-access-lc74l\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:20:58.635450 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.635445 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:20:58.723432 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.723350 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:58.840629 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:58.840597 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:20:59.415842 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:59.415816 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" Sep 11 19:20:59.416300 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:59.415857 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz" event={"ID":"8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1","Type":"ContainerDied","Data":"e0cbc91115ee77a60d93d26d0852566a63a3e377ae0547aef9276f0a05ed370f"} Sep 11 19:20:59.416300 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:59.415915 2556 scope.go:117] "RemoveContainer" containerID="4e5a06da80b88c0e5e9f29551bbdcb4b37a43912f113e0e4479d4b321d406c3c" Sep 11 19:20:59.432435 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:59.432372 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz"] Sep 11 19:20:59.438400 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:59.438377 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-021613-564789ccc5-2f8qz"] Sep 11 19:20:59.581330 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:20:59.581297 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:00.554469 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:00.554432 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:01.177558 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:01.177512 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1" path="/var/lib/kubelet/pods/8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1/volumes" Sep 11 19:21:01.266803 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:01.266766 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:03.802294 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:03.802263 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:40.534437 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:40.534406 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:42.224173 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:42.224120 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:46.397510 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:46.397472 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:46.495253 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:46.495214 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:46.660344 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:46.660261 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:47.284995 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:47.284956 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:47.987455 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:47.987421 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:48.826216 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:48.826179 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:49.543992 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:49.543962 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:50.255892 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:50.255852 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:57.510861 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:57.510817 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:57.535029 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:57.535001 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm"] Sep 11 19:21:57.535225 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:57.535200 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" podUID="493c5ad8-37e6-4e1a-a223-93eeb69c37d9" containerName="payload-processing" containerID="cri-o://4662d2feb26c8e3352a31413ceb8620b88a7f4c3c04fd41f361d5101cb2361bb" gracePeriod=30 Sep 11 19:21:57.611069 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:57.611017 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:57.709660 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:57.709631 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:57.717852 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:57.717823 2556 generic.go:356] "Generic (PLEG): container finished" podID="493c5ad8-37e6-4e1a-a223-93eeb69c37d9" containerID="4662d2feb26c8e3352a31413ceb8620b88a7f4c3c04fd41f361d5101cb2361bb" exitCode=0 Sep 11 19:21:57.717986 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:57.717896 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" event={"ID":"493c5ad8-37e6-4e1a-a223-93eeb69c37d9","Type":"ContainerDied","Data":"4662d2feb26c8e3352a31413ceb8620b88a7f4c3c04fd41f361d5101cb2361bb"} Sep 11 19:21:57.771854 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:57.771813 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:57.793047 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:57.793022 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" Sep 11 19:21:57.970827 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:57.970797 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/493c5ad8-37e6-4e1a-a223-93eeb69c37d9-plugins-config-volume\") pod \"493c5ad8-37e6-4e1a-a223-93eeb69c37d9\" (UID: \"493c5ad8-37e6-4e1a-a223-93eeb69c37d9\") " Sep 11 19:21:57.971002 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:57.970852 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfkxq\" (UniqueName: \"kubernetes.io/projected/493c5ad8-37e6-4e1a-a223-93eeb69c37d9-kube-api-access-vfkxq\") pod \"493c5ad8-37e6-4e1a-a223-93eeb69c37d9\" (UID: \"493c5ad8-37e6-4e1a-a223-93eeb69c37d9\") " Sep 11 19:21:57.971264 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:57.971230 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493c5ad8-37e6-4e1a-a223-93eeb69c37d9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "493c5ad8-37e6-4e1a-a223-93eeb69c37d9" (UID: "493c5ad8-37e6-4e1a-a223-93eeb69c37d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:21:57.972922 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:57.972901 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493c5ad8-37e6-4e1a-a223-93eeb69c37d9-kube-api-access-vfkxq" (OuterVolumeSpecName: "kube-api-access-vfkxq") pod "493c5ad8-37e6-4e1a-a223-93eeb69c37d9" (UID: "493c5ad8-37e6-4e1a-a223-93eeb69c37d9"). InnerVolumeSpecName "kube-api-access-vfkxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:21:58.072069 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:58.072027 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfkxq\" (UniqueName: \"kubernetes.io/projected/493c5ad8-37e6-4e1a-a223-93eeb69c37d9-kube-api-access-vfkxq\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:21:58.072069 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:58.072065 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/493c5ad8-37e6-4e1a-a223-93eeb69c37d9-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:21:58.084956 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:58.084927 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:58.183640 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:58.183605 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:58.724572 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:58.724530 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" Sep 11 19:21:58.724572 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:58.724561 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm" event={"ID":"493c5ad8-37e6-4e1a-a223-93eeb69c37d9","Type":"ContainerDied","Data":"a0ef3a56f3ee2f72378b0b8d243b7e0dcfa0d569973dcabbd4bb95e5e6ebac03"} Sep 11 19:21:58.725158 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:58.724634 2556 scope.go:117] "RemoveContainer" containerID="4662d2feb26c8e3352a31413ceb8620b88a7f4c3c04fd41f361d5101cb2361bb" Sep 11 19:21:58.745299 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:58.745277 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm"] Sep 11 19:21:58.747505 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:58.747483 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-afd85c-865fdbc7b5-j4ncm"] Sep 11 19:21:58.913415 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:58.913376 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:21:59.171617 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:59.171585 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="493c5ad8-37e6-4e1a-a223-93eeb69c37d9" path="/var/lib/kubelet/pods/493c5ad8-37e6-4e1a-a223-93eeb69c37d9/volumes" Sep 11 19:21:59.837260 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:21:59.837227 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:03.926446 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:03.926399 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:04.028808 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:04.028770 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:04.191674 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:04.191588 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:04.751319 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:04.751283 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:05.559679 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:05.559625 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:06.369764 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:06.369718 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:07.055384 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:07.055350 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:15.338233 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.338196 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd"] Sep 11 19:22:15.338908 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.338418 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" podUID="668f5591-b871-4427-ac57-f52e2f7e8aa6" containerName="maas-api" containerID="cri-o://41f1fbc9b8b863ccf3954dd617037d01436e8b20d732df49fd92ccd4f8bc4c80" gracePeriod=30 Sep 11 19:22:15.381826 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.381789 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:15.392611 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.392571 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8280eb-29819235-j6fv7"] Sep 11 19:22:15.398922 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.398892 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8280eb-29819235-j6fv7"] Sep 11 19:22:15.428178 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.428144 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp"] Sep 11 19:22:15.428410 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.428381 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" podUID="a0ea7ca2-b697-43a8-ba83-777729547927" containerName="payload-processing" containerID="cri-o://7ab7a88d6f12887aeed35851adc1656351fec4ed7779aec215603d5f174b72e5" gracePeriod=30 Sep 11 19:22:15.483495 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.483461 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:15.596693 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.596602 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:15.688141 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.688112 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:22:15.707819 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.707793 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" Sep 11 19:22:15.711929 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.711908 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:15.735525 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.735496 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668f5591-b871-4427-ac57-f52e2f7e8aa6-metrics-certs\") pod \"668f5591-b871-4427-ac57-f52e2f7e8aa6\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " Sep 11 19:22:15.735736 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.735591 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668f5591-b871-4427-ac57-f52e2f7e8aa6-openshift-service-ca\") pod \"668f5591-b871-4427-ac57-f52e2f7e8aa6\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " Sep 11 19:22:15.735736 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.735630 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxn9d\" (UniqueName: \"kubernetes.io/projected/a0ea7ca2-b697-43a8-ba83-777729547927-kube-api-access-kxn9d\") pod \"a0ea7ca2-b697-43a8-ba83-777729547927\" (UID: \"a0ea7ca2-b697-43a8-ba83-777729547927\") " Sep 11 19:22:15.735736 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.735659 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668f5591-b871-4427-ac57-f52e2f7e8aa6-maas-api-tls\") pod \"668f5591-b871-4427-ac57-f52e2f7e8aa6\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " Sep 11 19:22:15.735736 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.735689 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqcbq\" (UniqueName: \"kubernetes.io/projected/668f5591-b871-4427-ac57-f52e2f7e8aa6-kube-api-access-zqcbq\") pod \"668f5591-b871-4427-ac57-f52e2f7e8aa6\" (UID: \"668f5591-b871-4427-ac57-f52e2f7e8aa6\") " Sep 11 19:22:15.735736 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.735707 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ea7ca2-b697-43a8-ba83-777729547927-plugins-config-volume\") pod \"a0ea7ca2-b697-43a8-ba83-777729547927\" (UID: \"a0ea7ca2-b697-43a8-ba83-777729547927\") " Sep 11 19:22:15.736084 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.736012 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668f5591-b871-4427-ac57-f52e2f7e8aa6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "668f5591-b871-4427-ac57-f52e2f7e8aa6" (UID: "668f5591-b871-4427-ac57-f52e2f7e8aa6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:22:15.736512 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.736459 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ea7ca2-b697-43a8-ba83-777729547927-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a0ea7ca2-b697-43a8-ba83-777729547927" (UID: "a0ea7ca2-b697-43a8-ba83-777729547927"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:22:15.738212 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.738176 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668f5591-b871-4427-ac57-f52e2f7e8aa6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "668f5591-b871-4427-ac57-f52e2f7e8aa6" (UID: "668f5591-b871-4427-ac57-f52e2f7e8aa6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:22:15.738452 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.738427 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668f5591-b871-4427-ac57-f52e2f7e8aa6-kube-api-access-zqcbq" (OuterVolumeSpecName: "kube-api-access-zqcbq") pod "668f5591-b871-4427-ac57-f52e2f7e8aa6" (UID: "668f5591-b871-4427-ac57-f52e2f7e8aa6"). InnerVolumeSpecName "kube-api-access-zqcbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:22:15.738591 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.738496 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ea7ca2-b697-43a8-ba83-777729547927-kube-api-access-kxn9d" (OuterVolumeSpecName: "kube-api-access-kxn9d") pod "a0ea7ca2-b697-43a8-ba83-777729547927" (UID: "a0ea7ca2-b697-43a8-ba83-777729547927"). InnerVolumeSpecName "kube-api-access-kxn9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:22:15.739034 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.739009 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668f5591-b871-4427-ac57-f52e2f7e8aa6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "668f5591-b871-4427-ac57-f52e2f7e8aa6" (UID: "668f5591-b871-4427-ac57-f52e2f7e8aa6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:22:15.811238 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.811194 2556 generic.go:356] "Generic (PLEG): container finished" podID="a0ea7ca2-b697-43a8-ba83-777729547927" containerID="7ab7a88d6f12887aeed35851adc1656351fec4ed7779aec215603d5f174b72e5" exitCode=0 Sep 11 19:22:15.811238 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.811237 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" Sep 11 19:22:15.811488 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.811273 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" event={"ID":"a0ea7ca2-b697-43a8-ba83-777729547927","Type":"ContainerDied","Data":"7ab7a88d6f12887aeed35851adc1656351fec4ed7779aec215603d5f174b72e5"} Sep 11 19:22:15.811488 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.811324 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp" event={"ID":"a0ea7ca2-b697-43a8-ba83-777729547927","Type":"ContainerDied","Data":"9de877f2476fad9ab73d57fce6d6b60f93296be2cfd8edeb678834b82626cc0b"} Sep 11 19:22:15.811488 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.811350 2556 scope.go:117] "RemoveContainer" containerID="7ab7a88d6f12887aeed35851adc1656351fec4ed7779aec215603d5f174b72e5" Sep 11 19:22:15.812620 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.812592 2556 generic.go:356] "Generic (PLEG): container finished" podID="668f5591-b871-4427-ac57-f52e2f7e8aa6" containerID="41f1fbc9b8b863ccf3954dd617037d01436e8b20d732df49fd92ccd4f8bc4c80" exitCode=0 Sep 11 19:22:15.812732 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.812639 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" Sep 11 19:22:15.812732 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.812668 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" event={"ID":"668f5591-b871-4427-ac57-f52e2f7e8aa6","Type":"ContainerDied","Data":"41f1fbc9b8b863ccf3954dd617037d01436e8b20d732df49fd92ccd4f8bc4c80"} Sep 11 19:22:15.812732 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.812704 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd" event={"ID":"668f5591-b871-4427-ac57-f52e2f7e8aa6","Type":"ContainerDied","Data":"eba9eae41aac0b7cb904c50127b0776f5df1d31d1ad09851ae0ad64851674f58"} Sep 11 19:22:15.822590 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.815466 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:15.828104 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.828079 2556 scope.go:117] "RemoveContainer" containerID="7ab7a88d6f12887aeed35851adc1656351fec4ed7779aec215603d5f174b72e5" Sep 11 19:22:15.828406 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:22:15.828384 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab7a88d6f12887aeed35851adc1656351fec4ed7779aec215603d5f174b72e5\": container with ID starting with 7ab7a88d6f12887aeed35851adc1656351fec4ed7779aec215603d5f174b72e5 not found: ID does not exist" containerID="7ab7a88d6f12887aeed35851adc1656351fec4ed7779aec215603d5f174b72e5" Sep 11 19:22:15.828486 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.828415 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab7a88d6f12887aeed35851adc1656351fec4ed7779aec215603d5f174b72e5"} err="failed to get container status \"7ab7a88d6f12887aeed35851adc1656351fec4ed7779aec215603d5f174b72e5\": rpc error: code = NotFound desc = could not find container \"7ab7a88d6f12887aeed35851adc1656351fec4ed7779aec215603d5f174b72e5\": container with ID starting with 7ab7a88d6f12887aeed35851adc1656351fec4ed7779aec215603d5f174b72e5 not found: ID does not exist" Sep 11 19:22:15.828486 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.828431 2556 scope.go:117] "RemoveContainer" containerID="41f1fbc9b8b863ccf3954dd617037d01436e8b20d732df49fd92ccd4f8bc4c80" Sep 11 19:22:15.836401 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.836383 2556 scope.go:117] "RemoveContainer" containerID="41f1fbc9b8b863ccf3954dd617037d01436e8b20d732df49fd92ccd4f8bc4c80" Sep 11 19:22:15.836689 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:22:15.836669 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f1fbc9b8b863ccf3954dd617037d01436e8b20d732df49fd92ccd4f8bc4c80\": container with ID starting with 41f1fbc9b8b863ccf3954dd617037d01436e8b20d732df49fd92ccd4f8bc4c80 not found: ID does not exist" containerID="41f1fbc9b8b863ccf3954dd617037d01436e8b20d732df49fd92ccd4f8bc4c80" Sep 11 19:22:15.836757 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.836694 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f1fbc9b8b863ccf3954dd617037d01436e8b20d732df49fd92ccd4f8bc4c80"} err="failed to get container status \"41f1fbc9b8b863ccf3954dd617037d01436e8b20d732df49fd92ccd4f8bc4c80\": rpc error: code = NotFound desc = could not find container \"41f1fbc9b8b863ccf3954dd617037d01436e8b20d732df49fd92ccd4f8bc4c80\": container with ID starting with 41f1fbc9b8b863ccf3954dd617037d01436e8b20d732df49fd92ccd4f8bc4c80 not found: ID does not exist" Sep 11 19:22:15.836810 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.836759 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668f5591-b871-4427-ac57-f52e2f7e8aa6-openshift-service-ca\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:22:15.836810 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.836778 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxn9d\" (UniqueName: \"kubernetes.io/projected/a0ea7ca2-b697-43a8-ba83-777729547927-kube-api-access-kxn9d\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:22:15.836810 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.836788 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668f5591-b871-4427-ac57-f52e2f7e8aa6-maas-api-tls\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:22:15.836810 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.836796 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqcbq\" (UniqueName: \"kubernetes.io/projected/668f5591-b871-4427-ac57-f52e2f7e8aa6-kube-api-access-zqcbq\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:22:15.836810 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.836804 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ea7ca2-b697-43a8-ba83-777729547927-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:22:15.836810 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.836811 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668f5591-b871-4427-ac57-f52e2f7e8aa6-metrics-certs\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:22:15.843859 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.843795 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp"] Sep 11 19:22:15.846450 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.846428 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8280eb-555c875689-k69qp"] Sep 11 19:22:15.866417 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.866379 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd"] Sep 11 19:22:15.869122 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:15.869097 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8280eb-6df4f4768-v2dtd"] Sep 11 19:22:16.078885 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:16.078848 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:16.179328 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:16.179229 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:16.949932 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:16.949891 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:17.172321 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:17.172286 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="668f5591-b871-4427-ac57-f52e2f7e8aa6" path="/var/lib/kubelet/pods/668f5591-b871-4427-ac57-f52e2f7e8aa6/volumes" Sep 11 19:22:17.172695 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:17.172679 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0ea7ca2-b697-43a8-ba83-777729547927" path="/var/lib/kubelet/pods/a0ea7ca2-b697-43a8-ba83-777729547927/volumes" Sep 11 19:22:17.172945 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:17.172933 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7c95bf5-8d14-418c-b236-792e71442b33" path="/var/lib/kubelet/pods/b7c95bf5-8d14-418c-b236-792e71442b33/volumes" Sep 11 19:22:22.964970 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:22.964936 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:23.069319 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.069288 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:23.257159 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.257073 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:23.592192 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.592155 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb"] Sep 11 19:22:23.592565 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.592535 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="668f5591-b871-4427-ac57-f52e2f7e8aa6" containerName="maas-api" Sep 11 19:22:23.592565 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.592563 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="668f5591-b871-4427-ac57-f52e2f7e8aa6" containerName="maas-api" Sep 11 19:22:23.592664 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.592574 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1" containerName="payload-processing" Sep 11 19:22:23.592664 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.592580 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1" containerName="payload-processing" Sep 11 19:22:23.592664 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.592591 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ea7ca2-b697-43a8-ba83-777729547927" containerName="payload-processing" Sep 11 19:22:23.592664 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.592596 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ea7ca2-b697-43a8-ba83-777729547927" containerName="payload-processing" Sep 11 19:22:23.592664 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.592614 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493c5ad8-37e6-4e1a-a223-93eeb69c37d9" containerName="payload-processing" Sep 11 19:22:23.592664 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.592619 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="493c5ad8-37e6-4e1a-a223-93eeb69c37d9" containerName="payload-processing" Sep 11 19:22:23.592664 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.592626 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24509d56-8ec0-4f45-995a-846cd599146b" containerName="maas-api" Sep 11 19:22:23.592664 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.592630 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="24509d56-8ec0-4f45-995a-846cd599146b" containerName="maas-api" Sep 11 19:22:23.592956 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.592695 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ea7ca2-b697-43a8-ba83-777729547927" containerName="payload-processing" Sep 11 19:22:23.592956 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.592702 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="668f5591-b871-4427-ac57-f52e2f7e8aa6" containerName="maas-api" Sep 11 19:22:23.592956 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.592710 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eb5baaa-6c60-4158-bc86-40fd1a6b0cd1" containerName="payload-processing" Sep 11 19:22:23.592956 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.592717 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="24509d56-8ec0-4f45-995a-846cd599146b" containerName="maas-api" Sep 11 19:22:23.592956 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.592723 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="493c5ad8-37e6-4e1a-a223-93eeb69c37d9" containerName="payload-processing" Sep 11 19:22:23.595752 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.595737 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb" Sep 11 19:22:23.599135 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.599116 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ln295\"" Sep 11 19:22:23.606921 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.606899 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb"] Sep 11 19:22:23.710367 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.710333 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6051caf9-242a-4533-aaff-df62078db3e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb\" (UID: \"6051caf9-242a-4533-aaff-df62078db3e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb" Sep 11 19:22:23.811397 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.811361 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6051caf9-242a-4533-aaff-df62078db3e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb\" (UID: \"6051caf9-242a-4533-aaff-df62078db3e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb" Sep 11 19:22:23.812046 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.812028 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6051caf9-242a-4533-aaff-df62078db3e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb\" (UID: \"6051caf9-242a-4533-aaff-df62078db3e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb" Sep 11 19:22:23.905944 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:23.905859 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb" Sep 11 19:22:24.182419 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:24.182252 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb"] Sep 11 19:22:24.185057 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:22:24.185029 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6051caf9_242a_4533_aaff_df62078db3e0.slice/crio-aef9685ddb86553efe99074d17d5bcde757bbfb6ee9c62c93769f4344bd433a0 WatchSource:0}: Error finding container aef9685ddb86553efe99074d17d5bcde757bbfb6ee9c62c93769f4344bd433a0: Status 404 returned error can't find the container with id aef9685ddb86553efe99074d17d5bcde757bbfb6ee9c62c93769f4344bd433a0 Sep 11 19:22:24.566472 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:24.566428 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:24.862764 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:24.862728 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb" event={"ID":"6051caf9-242a-4533-aaff-df62078db3e0","Type":"ContainerStarted","Data":"d97de469576f6b5d982ec6f0db6b48bf9e285f4fecf66d511ba548eaa2d87e07"} Sep 11 19:22:24.862937 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:24.862791 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb" event={"ID":"6051caf9-242a-4533-aaff-df62078db3e0","Type":"ContainerStarted","Data":"aef9685ddb86553efe99074d17d5bcde757bbfb6ee9c62c93769f4344bd433a0"} Sep 11 19:22:24.884080 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:24.884026 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb" podStartSLOduration=1.884006108 podStartE2EDuration="1.884006108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:22:24.882328133 +0000 UTC m=+2050.336134107" watchObservedRunningTime="2026-09-11 19:22:24.884006108 +0000 UTC m=+2050.337812065" Sep 11 19:22:25.403626 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:25.403597 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:25.869430 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:25.869396 2556 generic.go:356] "Generic (PLEG): container finished" podID="6051caf9-242a-4533-aaff-df62078db3e0" containerID="d97de469576f6b5d982ec6f0db6b48bf9e285f4fecf66d511ba548eaa2d87e07" exitCode=0 Sep 11 19:22:25.869617 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:25.869484 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb" event={"ID":"6051caf9-242a-4533-aaff-df62078db3e0","Type":"ContainerDied","Data":"d97de469576f6b5d982ec6f0db6b48bf9e285f4fecf66d511ba548eaa2d87e07"} Sep 11 19:22:27.005204 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:27.005179 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb" Sep 11 19:22:27.142002 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:27.141928 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6051caf9-242a-4533-aaff-df62078db3e0-maas-api-service-ca\") pod \"6051caf9-242a-4533-aaff-df62078db3e0\" (UID: \"6051caf9-242a-4533-aaff-df62078db3e0\") " Sep 11 19:22:27.142284 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:27.142257 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6051caf9-242a-4533-aaff-df62078db3e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6051caf9-242a-4533-aaff-df62078db3e0" (UID: "6051caf9-242a-4533-aaff-df62078db3e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:22:27.243114 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:27.243089 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6051caf9-242a-4533-aaff-df62078db3e0-maas-api-service-ca\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:22:27.881367 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:27.881334 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb" Sep 11 19:22:27.881555 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:27.881368 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb" event={"ID":"6051caf9-242a-4533-aaff-df62078db3e0","Type":"ContainerDied","Data":"aef9685ddb86553efe99074d17d5bcde757bbfb6ee9c62c93769f4344bd433a0"} Sep 11 19:22:27.881555 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:27.881421 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aef9685ddb86553efe99074d17d5bcde757bbfb6ee9c62c93769f4344bd433a0" Sep 11 19:22:34.222111 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.222034 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:34.261232 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.261179 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z"] Sep 11 19:22:34.262101 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.262064 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" podUID="7097a4f9-7ad3-4910-b2ff-f206f17fa531" containerName="payload-processing" containerID="cri-o://d73652dc6c63d196cef2c2f3d1c4d4a9fb5ab160530d97d05a6e24a161e57016" gracePeriod=30 Sep 11 19:22:34.270691 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.270663 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh"] Sep 11 19:22:34.270946 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.270913 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" podUID="936bb385-4848-4cf3-b33c-0a4366114713" containerName="payload-pre-processing" containerID="cri-o://168f68e8689a64e2a6a5d95547258506c614978cdc2a88f97b2fa7476e0765aa" gracePeriod=30 Sep 11 19:22:34.304580 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.304517 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:34.375244 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.375210 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:34.484486 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.484418 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:34.518911 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.518883 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb"] Sep 11 19:22:34.521402 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.521380 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3b7391-pn6cb"] Sep 11 19:22:34.529427 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.529406 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" Sep 11 19:22:34.563249 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.563219 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" Sep 11 19:22:34.599907 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.599858 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:34.609817 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.609790 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7xm2\" (UniqueName: \"kubernetes.io/projected/7097a4f9-7ad3-4910-b2ff-f206f17fa531-kube-api-access-l7xm2\") pod \"7097a4f9-7ad3-4910-b2ff-f206f17fa531\" (UID: \"7097a4f9-7ad3-4910-b2ff-f206f17fa531\") " Sep 11 19:22:34.610008 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.609835 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7097a4f9-7ad3-4910-b2ff-f206f17fa531-plugins-config-volume\") pod \"7097a4f9-7ad3-4910-b2ff-f206f17fa531\" (UID: \"7097a4f9-7ad3-4910-b2ff-f206f17fa531\") " Sep 11 19:22:34.610251 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.610214 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7097a4f9-7ad3-4910-b2ff-f206f17fa531-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7097a4f9-7ad3-4910-b2ff-f206f17fa531" (UID: "7097a4f9-7ad3-4910-b2ff-f206f17fa531"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:22:34.612018 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.611997 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7097a4f9-7ad3-4910-b2ff-f206f17fa531-kube-api-access-l7xm2" (OuterVolumeSpecName: "kube-api-access-l7xm2") pod "7097a4f9-7ad3-4910-b2ff-f206f17fa531" (UID: "7097a4f9-7ad3-4910-b2ff-f206f17fa531"). InnerVolumeSpecName "kube-api-access-l7xm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:22:34.710676 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.710642 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2qw8\" (UniqueName: \"kubernetes.io/projected/936bb385-4848-4cf3-b33c-0a4366114713-kube-api-access-q2qw8\") pod \"936bb385-4848-4cf3-b33c-0a4366114713\" (UID: \"936bb385-4848-4cf3-b33c-0a4366114713\") " Sep 11 19:22:34.710869 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.710731 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936bb385-4848-4cf3-b33c-0a4366114713-plugins-config-volume\") pod \"936bb385-4848-4cf3-b33c-0a4366114713\" (UID: \"936bb385-4848-4cf3-b33c-0a4366114713\") " Sep 11 19:22:34.710941 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.710904 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7xm2\" (UniqueName: \"kubernetes.io/projected/7097a4f9-7ad3-4910-b2ff-f206f17fa531-kube-api-access-l7xm2\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:22:34.710941 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.710916 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7097a4f9-7ad3-4910-b2ff-f206f17fa531-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:22:34.711099 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.711074 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936bb385-4848-4cf3-b33c-0a4366114713-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "936bb385-4848-4cf3-b33c-0a4366114713" (UID: "936bb385-4848-4cf3-b33c-0a4366114713"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:22:34.712785 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.712762 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936bb385-4848-4cf3-b33c-0a4366114713-kube-api-access-q2qw8" (OuterVolumeSpecName: "kube-api-access-q2qw8") pod "936bb385-4848-4cf3-b33c-0a4366114713" (UID: "936bb385-4848-4cf3-b33c-0a4366114713"). InnerVolumeSpecName "kube-api-access-q2qw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:22:34.811518 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.811476 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2qw8\" (UniqueName: \"kubernetes.io/projected/936bb385-4848-4cf3-b33c-0a4366114713-kube-api-access-q2qw8\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:22:34.811518 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.811505 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936bb385-4848-4cf3-b33c-0a4366114713-plugins-config-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:22:34.916032 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.915997 2556 generic.go:356] "Generic (PLEG): container finished" podID="7097a4f9-7ad3-4910-b2ff-f206f17fa531" containerID="d73652dc6c63d196cef2c2f3d1c4d4a9fb5ab160530d97d05a6e24a161e57016" exitCode=0 Sep 11 19:22:34.916228 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.916049 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" Sep 11 19:22:34.916228 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.916086 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" event={"ID":"7097a4f9-7ad3-4910-b2ff-f206f17fa531","Type":"ContainerDied","Data":"d73652dc6c63d196cef2c2f3d1c4d4a9fb5ab160530d97d05a6e24a161e57016"} Sep 11 19:22:34.916228 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.916139 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z" event={"ID":"7097a4f9-7ad3-4910-b2ff-f206f17fa531","Type":"ContainerDied","Data":"0b6de8a2b4f05eca4ae8cf219b5528f47964b19b995ac4a13bda871e1bef9615"} Sep 11 19:22:34.916228 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.916166 2556 scope.go:117] "RemoveContainer" containerID="d73652dc6c63d196cef2c2f3d1c4d4a9fb5ab160530d97d05a6e24a161e57016" Sep 11 19:22:34.917571 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.917518 2556 generic.go:356] "Generic (PLEG): container finished" podID="936bb385-4848-4cf3-b33c-0a4366114713" containerID="168f68e8689a64e2a6a5d95547258506c614978cdc2a88f97b2fa7476e0765aa" exitCode=0 Sep 11 19:22:34.917688 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.917568 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" event={"ID":"936bb385-4848-4cf3-b33c-0a4366114713","Type":"ContainerDied","Data":"168f68e8689a64e2a6a5d95547258506c614978cdc2a88f97b2fa7476e0765aa"} Sep 11 19:22:34.917688 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.917618 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" event={"ID":"936bb385-4848-4cf3-b33c-0a4366114713","Type":"ContainerDied","Data":"4a63d259d89319e8e12f95b04b2dfefe4d87cf045fb4a6adc27b5f89a1c87bc0"} Sep 11 19:22:34.917688 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.917574 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh" Sep 11 19:22:34.927249 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.927226 2556 scope.go:117] "RemoveContainer" containerID="d73652dc6c63d196cef2c2f3d1c4d4a9fb5ab160530d97d05a6e24a161e57016" Sep 11 19:22:34.927626 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:22:34.927597 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d73652dc6c63d196cef2c2f3d1c4d4a9fb5ab160530d97d05a6e24a161e57016\": container with ID starting with d73652dc6c63d196cef2c2f3d1c4d4a9fb5ab160530d97d05a6e24a161e57016 not found: ID does not exist" containerID="d73652dc6c63d196cef2c2f3d1c4d4a9fb5ab160530d97d05a6e24a161e57016" Sep 11 19:22:34.927727 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.927635 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73652dc6c63d196cef2c2f3d1c4d4a9fb5ab160530d97d05a6e24a161e57016"} err="failed to get container status \"d73652dc6c63d196cef2c2f3d1c4d4a9fb5ab160530d97d05a6e24a161e57016\": rpc error: code = NotFound desc = could not find container \"d73652dc6c63d196cef2c2f3d1c4d4a9fb5ab160530d97d05a6e24a161e57016\": container with ID starting with d73652dc6c63d196cef2c2f3d1c4d4a9fb5ab160530d97d05a6e24a161e57016 not found: ID does not exist" Sep 11 19:22:34.927727 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.927658 2556 scope.go:117] "RemoveContainer" containerID="168f68e8689a64e2a6a5d95547258506c614978cdc2a88f97b2fa7476e0765aa" Sep 11 19:22:34.928330 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.928288 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:34.936286 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.936269 2556 scope.go:117] "RemoveContainer" containerID="168f68e8689a64e2a6a5d95547258506c614978cdc2a88f97b2fa7476e0765aa" Sep 11 19:22:34.936564 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:22:34.936527 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168f68e8689a64e2a6a5d95547258506c614978cdc2a88f97b2fa7476e0765aa\": container with ID starting with 168f68e8689a64e2a6a5d95547258506c614978cdc2a88f97b2fa7476e0765aa not found: ID does not exist" containerID="168f68e8689a64e2a6a5d95547258506c614978cdc2a88f97b2fa7476e0765aa" Sep 11 19:22:34.936638 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.936568 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168f68e8689a64e2a6a5d95547258506c614978cdc2a88f97b2fa7476e0765aa"} err="failed to get container status \"168f68e8689a64e2a6a5d95547258506c614978cdc2a88f97b2fa7476e0765aa\": rpc error: code = NotFound desc = could not find container \"168f68e8689a64e2a6a5d95547258506c614978cdc2a88f97b2fa7476e0765aa\": container with ID starting with 168f68e8689a64e2a6a5d95547258506c614978cdc2a88f97b2fa7476e0765aa not found: ID does not exist" Sep 11 19:22:34.946526 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.946504 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z"] Sep 11 19:22:34.953123 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.953097 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b7391-657b9d7cd7-skk9z"] Sep 11 19:22:34.962987 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.962962 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh"] Sep 11 19:22:34.967976 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:34.967954 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3b7391-547f4697f4-gmxlh"] Sep 11 19:22:35.020311 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:35.020274 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:35.175123 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:35.175041 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6051caf9-242a-4533-aaff-df62078db3e0" path="/var/lib/kubelet/pods/6051caf9-242a-4533-aaff-df62078db3e0/volumes" Sep 11 19:22:35.175425 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:35.175411 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7097a4f9-7ad3-4910-b2ff-f206f17fa531" path="/var/lib/kubelet/pods/7097a4f9-7ad3-4910-b2ff-f206f17fa531/volumes" Sep 11 19:22:35.175775 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:35.175762 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="936bb385-4848-4cf3-b33c-0a4366114713" path="/var/lib/kubelet/pods/936bb385-4848-4cf3-b33c-0a4366114713/volumes" Sep 11 19:22:41.657994 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:41.657960 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:41.768974 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:41.768937 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:41.940978 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:41.940887 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:22:53.187272 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:22:53.187198 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:23:15.193039 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:23:15.193015 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 19:23:15.198900 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:23:15.198880 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 19:23:15.537166 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:23:15.537134 2556 scope.go:117] "RemoveContainer" containerID="c8e6f93a8633e5e3d0fab39a94c014c488017accbb43e3a3edf36b20b2f53012" Sep 11 19:24:08.395216 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:24:08.395121 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:24:10.014891 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:24:10.014854 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:24:25.775790 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:24:25.775755 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:24:30.032336 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:24:30.032303 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:24:51.280677 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:24:51.280625 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:25:01.599228 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:01.599193 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:25:36.160727 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.160686 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh"] Sep 11 19:25:36.161338 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.160958 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" podUID="a09e5f0f-c582-4d68-a7b5-b8599e41b283" containerName="manager" containerID="cri-o://7ab679f88706cca4e170178157bf6d818904f444e290652f3dd6c7d13ead7464" gracePeriod=10 Sep 11 19:25:36.412924 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.412861 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" Sep 11 19:25:36.420210 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.420191 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a09e5f0f-c582-4d68-a7b5-b8599e41b283-extensions-socket-volume\") pod \"a09e5f0f-c582-4d68-a7b5-b8599e41b283\" (UID: \"a09e5f0f-c582-4d68-a7b5-b8599e41b283\") " Sep 11 19:25:36.420342 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.420252 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmxx2\" (UniqueName: \"kubernetes.io/projected/a09e5f0f-c582-4d68-a7b5-b8599e41b283-kube-api-access-pmxx2\") pod \"a09e5f0f-c582-4d68-a7b5-b8599e41b283\" (UID: \"a09e5f0f-c582-4d68-a7b5-b8599e41b283\") " Sep 11 19:25:36.420604 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.420578 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09e5f0f-c582-4d68-a7b5-b8599e41b283-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a09e5f0f-c582-4d68-a7b5-b8599e41b283" (UID: "a09e5f0f-c582-4d68-a7b5-b8599e41b283"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:25:36.422355 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.422335 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09e5f0f-c582-4d68-a7b5-b8599e41b283-kube-api-access-pmxx2" (OuterVolumeSpecName: "kube-api-access-pmxx2") pod "a09e5f0f-c582-4d68-a7b5-b8599e41b283" (UID: "a09e5f0f-c582-4d68-a7b5-b8599e41b283"). InnerVolumeSpecName "kube-api-access-pmxx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:25:36.521710 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.521674 2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a09e5f0f-c582-4d68-a7b5-b8599e41b283-extensions-socket-volume\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:25:36.521710 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.521705 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmxx2\" (UniqueName: \"kubernetes.io/projected/a09e5f0f-c582-4d68-a7b5-b8599e41b283-kube-api-access-pmxx2\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:25:36.814244 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.814212 2556 generic.go:356] "Generic (PLEG): container finished" podID="a09e5f0f-c582-4d68-a7b5-b8599e41b283" containerID="7ab679f88706cca4e170178157bf6d818904f444e290652f3dd6c7d13ead7464" exitCode=0 Sep 11 19:25:36.814425 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.814263 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" Sep 11 19:25:36.814425 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.814280 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" event={"ID":"a09e5f0f-c582-4d68-a7b5-b8599e41b283","Type":"ContainerDied","Data":"7ab679f88706cca4e170178157bf6d818904f444e290652f3dd6c7d13ead7464"} Sep 11 19:25:36.814425 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.814320 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh" event={"ID":"a09e5f0f-c582-4d68-a7b5-b8599e41b283","Type":"ContainerDied","Data":"08d751ec6b29f2388bde37b146a4909a50d36b2cd404b72f0e49dfa6943d8c33"} Sep 11 19:25:36.814425 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.814340 2556 scope.go:117] "RemoveContainer" containerID="7ab679f88706cca4e170178157bf6d818904f444e290652f3dd6c7d13ead7464" Sep 11 19:25:36.823702 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.823683 2556 scope.go:117] "RemoveContainer" containerID="7ab679f88706cca4e170178157bf6d818904f444e290652f3dd6c7d13ead7464" Sep 11 19:25:36.823975 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:25:36.823948 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab679f88706cca4e170178157bf6d818904f444e290652f3dd6c7d13ead7464\": container with ID starting with 7ab679f88706cca4e170178157bf6d818904f444e290652f3dd6c7d13ead7464 not found: ID does not exist" containerID="7ab679f88706cca4e170178157bf6d818904f444e290652f3dd6c7d13ead7464" Sep 11 19:25:36.824039 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.823983 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab679f88706cca4e170178157bf6d818904f444e290652f3dd6c7d13ead7464"} err="failed to get container status \"7ab679f88706cca4e170178157bf6d818904f444e290652f3dd6c7d13ead7464\": rpc error: code = NotFound desc = could not find container \"7ab679f88706cca4e170178157bf6d818904f444e290652f3dd6c7d13ead7464\": container with ID starting with 7ab679f88706cca4e170178157bf6d818904f444e290652f3dd6c7d13ead7464 not found: ID does not exist" Sep 11 19:25:36.834899 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.834878 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh"] Sep 11 19:25:36.838477 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:36.838455 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-frrkh"] Sep 11 19:25:37.171715 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:25:37.171609 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a09e5f0f-c582-4d68-a7b5-b8599e41b283" path="/var/lib/kubelet/pods/a09e5f0f-c582-4d68-a7b5-b8599e41b283/volumes" Sep 11 19:26:36.850903 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.850869 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fjcfw"] Sep 11 19:26:36.851315 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.851268 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7097a4f9-7ad3-4910-b2ff-f206f17fa531" containerName="payload-processing" Sep 11 19:26:36.851315 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.851281 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7097a4f9-7ad3-4910-b2ff-f206f17fa531" containerName="payload-processing" Sep 11 19:26:36.851315 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.851293 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09e5f0f-c582-4d68-a7b5-b8599e41b283" containerName="manager" Sep 11 19:26:36.851315 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.851298 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09e5f0f-c582-4d68-a7b5-b8599e41b283" containerName="manager" Sep 11 19:26:36.851315 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.851305 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6051caf9-242a-4533-aaff-df62078db3e0" containerName="revoke-keys" Sep 11 19:26:36.851315 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.851310 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6051caf9-242a-4533-aaff-df62078db3e0" containerName="revoke-keys" Sep 11 19:26:36.851315 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.851318 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="936bb385-4848-4cf3-b33c-0a4366114713" containerName="payload-pre-processing" Sep 11 19:26:36.851532 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.851323 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="936bb385-4848-4cf3-b33c-0a4366114713" containerName="payload-pre-processing" Sep 11 19:26:36.851532 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.851387 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6051caf9-242a-4533-aaff-df62078db3e0" containerName="revoke-keys" Sep 11 19:26:36.851532 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.851396 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a09e5f0f-c582-4d68-a7b5-b8599e41b283" containerName="manager" Sep 11 19:26:36.851532 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.851403 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="936bb385-4848-4cf3-b33c-0a4366114713" containerName="payload-pre-processing" Sep 11 19:26:36.851532 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.851409 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7097a4f9-7ad3-4910-b2ff-f206f17fa531" containerName="payload-processing" Sep 11 19:26:36.854342 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.854325 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fjcfw" Sep 11 19:26:36.856816 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.856788 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r9t7x\"" Sep 11 19:26:36.856955 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.856865 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 19:26:36.856955 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.856899 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 19:26:36.865103 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.865079 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fjcfw"] Sep 11 19:26:36.969881 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.969848 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8650c92-d57f-4877-b214-b58cefdc073c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fjcfw\" (UID: \"c8650c92-d57f-4877-b214-b58cefdc073c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fjcfw" Sep 11 19:26:36.970040 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:36.969889 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkxl7\" (UniqueName: \"kubernetes.io/projected/c8650c92-d57f-4877-b214-b58cefdc073c-kube-api-access-zkxl7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fjcfw\" (UID: \"c8650c92-d57f-4877-b214-b58cefdc073c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fjcfw" Sep 11 19:26:37.070640 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:37.070606 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8650c92-d57f-4877-b214-b58cefdc073c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fjcfw\" (UID: \"c8650c92-d57f-4877-b214-b58cefdc073c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fjcfw" Sep 11 19:26:37.070761 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:37.070644 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkxl7\" (UniqueName: \"kubernetes.io/projected/c8650c92-d57f-4877-b214-b58cefdc073c-kube-api-access-zkxl7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fjcfw\" (UID: \"c8650c92-d57f-4877-b214-b58cefdc073c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fjcfw" Sep 11 19:26:37.071049 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:37.071028 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8650c92-d57f-4877-b214-b58cefdc073c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fjcfw\" (UID: \"c8650c92-d57f-4877-b214-b58cefdc073c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fjcfw" Sep 11 19:26:37.084506 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:37.084481 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkxl7\" (UniqueName: \"kubernetes.io/projected/c8650c92-d57f-4877-b214-b58cefdc073c-kube-api-access-zkxl7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fjcfw\" (UID: \"c8650c92-d57f-4877-b214-b58cefdc073c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fjcfw" Sep 11 19:26:37.165597 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:37.165485 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fjcfw" Sep 11 19:26:37.445227 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:37.445153 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fjcfw"] Sep 11 19:26:37.448533 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:26:37.448506 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8650c92_d57f_4877_b214_b58cefdc073c.slice/crio-003c3c561099dc4c6d638af3ba0728d758595b164dc26e103ccb605c33227856 WatchSource:0}: Error finding container 003c3c561099dc4c6d638af3ba0728d758595b164dc26e103ccb605c33227856: Status 404 returned error can't find the container with id 003c3c561099dc4c6d638af3ba0728d758595b164dc26e103ccb605c33227856 Sep 11 19:26:38.105257 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:38.105220 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fjcfw" event={"ID":"c8650c92-d57f-4877-b214-b58cefdc073c","Type":"ContainerStarted","Data":"211959892ca118033e701859c4e187b5e914018aec0fed1023909a4e370ab0f6"} Sep 11 19:26:38.105685 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:38.105270 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fjcfw" event={"ID":"c8650c92-d57f-4877-b214-b58cefdc073c","Type":"ContainerStarted","Data":"003c3c561099dc4c6d638af3ba0728d758595b164dc26e103ccb605c33227856"} Sep 11 19:26:38.105685 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:38.105369 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fjcfw" Sep 11 19:26:38.124200 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:38.124158 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fjcfw" podStartSLOduration=2.124146414 podStartE2EDuration="2.124146414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:38.122999587 +0000 UTC m=+2303.576805562" watchObservedRunningTime="2026-09-11 19:26:38.124146414 +0000 UTC m=+2303.577952367" Sep 11 19:26:49.111860 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:26:49.111832 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fjcfw" Sep 11 19:27:19.999214 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:27:19.999131 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:27:23.056323 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:27:23.056290 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:27:24.453159 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:27:24.453113 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:27:51.367002 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:27:51.366962 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:27:53.032181 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:27:53.032138 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:28:02.272846 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:02.272808 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:28:14.116210 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:14.116180 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:28:15.223108 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:15.223078 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 19:28:15.229155 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:15.229135 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42nvj_70e064d0-8173-4c6e-b32f-a903dbad84cc/ovn-acl-logging/0.log" Sep 11 19:28:16.145447 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.145416 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z"] Sep 11 19:28:16.148838 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.148819 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:16.151586 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.151563 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6bftc\"" Sep 11 19:28:16.151699 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.151566 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 19:28:16.151699 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.151602 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 19:28:16.159461 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.159437 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z"] Sep 11 19:28:16.263617 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.263588 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-metrics-certs\") pod \"maas-api-79856dc5dd-zqz6z\" (UID: \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\") " pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:16.263978 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.263622 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-maas-api-tls\") pod \"maas-api-79856dc5dd-zqz6z\" (UID: \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\") " pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:16.263978 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.263645 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-openshift-service-ca\") pod \"maas-api-79856dc5dd-zqz6z\" (UID: \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\") " pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:16.263978 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.263668 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkvgw\" (UniqueName: \"kubernetes.io/projected/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-kube-api-access-hkvgw\") pod \"maas-api-79856dc5dd-zqz6z\" (UID: \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\") " pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:16.365064 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.365021 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkvgw\" (UniqueName: \"kubernetes.io/projected/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-kube-api-access-hkvgw\") pod \"maas-api-79856dc5dd-zqz6z\" (UID: \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\") " pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:16.365245 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.365100 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-metrics-certs\") pod \"maas-api-79856dc5dd-zqz6z\" (UID: \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\") " pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:16.365312 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.365248 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-maas-api-tls\") pod \"maas-api-79856dc5dd-zqz6z\" (UID: \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\") " pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:16.365312 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.365294 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-openshift-service-ca\") pod \"maas-api-79856dc5dd-zqz6z\" (UID: \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\") " pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:16.365920 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.365894 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-openshift-service-ca\") pod \"maas-api-79856dc5dd-zqz6z\" (UID: \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\") " pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:16.367432 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.367411 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-metrics-certs\") pod \"maas-api-79856dc5dd-zqz6z\" (UID: \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\") " pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:16.367555 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.367460 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-maas-api-tls\") pod \"maas-api-79856dc5dd-zqz6z\" (UID: \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\") " pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:16.372615 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.372596 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkvgw\" (UniqueName: \"kubernetes.io/projected/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-kube-api-access-hkvgw\") pod \"maas-api-79856dc5dd-zqz6z\" (UID: \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\") " pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:16.462524 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.462460 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:16.734337 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.734283 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z"] Sep 11 19:28:16.736688 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:28:16.736659 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1328a435_9c84_4fc7_9140_bdf4f5ee41d2.slice/crio-c3f67d601117f3424a6e5305e151eede4e3f3f7fe0226c5a84be482a78d0a283 WatchSource:0}: Error finding container c3f67d601117f3424a6e5305e151eede4e3f3f7fe0226c5a84be482a78d0a283: Status 404 returned error can't find the container with id c3f67d601117f3424a6e5305e151eede4e3f3f7fe0226c5a84be482a78d0a283 Sep 11 19:28:16.738397 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:16.738379 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 19:28:17.583831 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:17.583794 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" event={"ID":"1328a435-9c84-4fc7-9140-bdf4f5ee41d2","Type":"ContainerStarted","Data":"5cc3551b7d1fdfb343aa463f5d6c21ff39090ff8c70966af352dd0ef75c982bd"} Sep 11 19:28:17.584331 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:17.583877 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" event={"ID":"1328a435-9c84-4fc7-9140-bdf4f5ee41d2","Type":"ContainerStarted","Data":"c3f67d601117f3424a6e5305e151eede4e3f3f7fe0226c5a84be482a78d0a283"} Sep 11 19:28:17.584331 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:17.583914 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:17.601473 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:17.601416 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" podStartSLOduration=1.2378934799999999 podStartE2EDuration="1.601399362s" totalImagesPullingTime="363.505882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:28:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:28:17.599846316 +0000 UTC m=+2403.053652272" watchObservedRunningTime="2026-09-11 19:28:17.601399362 +0000 UTC m=+2403.055205316" Sep 11 19:28:23.595148 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:23.595113 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:23.851855 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:23.851777 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n"] Sep 11 19:28:23.859359 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:23.859333 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" Sep 11 19:28:23.863987 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:23.863956 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n"] Sep 11 19:28:23.936925 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:23.936895 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mfds\" (UniqueName: \"kubernetes.io/projected/d8038bec-aa91-4c4b-b7a0-69e655bf31f5-kube-api-access-2mfds\") pod \"maas-api-79c474dfdd-p6w4n\" (UID: \"d8038bec-aa91-4c4b-b7a0-69e655bf31f5\") " pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" Sep 11 19:28:23.937075 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:23.936954 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8038bec-aa91-4c4b-b7a0-69e655bf31f5-metrics-certs\") pod \"maas-api-79c474dfdd-p6w4n\" (UID: \"d8038bec-aa91-4c4b-b7a0-69e655bf31f5\") " pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" Sep 11 19:28:23.937075 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:23.937043 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8038bec-aa91-4c4b-b7a0-69e655bf31f5-openshift-service-ca\") pod \"maas-api-79c474dfdd-p6w4n\" (UID: \"d8038bec-aa91-4c4b-b7a0-69e655bf31f5\") " pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" Sep 11 19:28:23.937161 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:23.937119 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8038bec-aa91-4c4b-b7a0-69e655bf31f5-maas-api-tls\") pod \"maas-api-79c474dfdd-p6w4n\" (UID: \"d8038bec-aa91-4c4b-b7a0-69e655bf31f5\") " pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" Sep 11 19:28:24.038214 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:24.038168 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8038bec-aa91-4c4b-b7a0-69e655bf31f5-maas-api-tls\") pod \"maas-api-79c474dfdd-p6w4n\" (UID: \"d8038bec-aa91-4c4b-b7a0-69e655bf31f5\") " pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" Sep 11 19:28:24.038214 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:24.038215 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mfds\" (UniqueName: \"kubernetes.io/projected/d8038bec-aa91-4c4b-b7a0-69e655bf31f5-kube-api-access-2mfds\") pod \"maas-api-79c474dfdd-p6w4n\" (UID: \"d8038bec-aa91-4c4b-b7a0-69e655bf31f5\") " pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" Sep 11 19:28:24.038413 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:24.038259 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8038bec-aa91-4c4b-b7a0-69e655bf31f5-metrics-certs\") pod \"maas-api-79c474dfdd-p6w4n\" (UID: \"d8038bec-aa91-4c4b-b7a0-69e655bf31f5\") " pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" Sep 11 19:28:24.038413 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:24.038292 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8038bec-aa91-4c4b-b7a0-69e655bf31f5-openshift-service-ca\") pod \"maas-api-79c474dfdd-p6w4n\" (UID: \"d8038bec-aa91-4c4b-b7a0-69e655bf31f5\") " pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" Sep 11 19:28:24.038969 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:24.038942 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8038bec-aa91-4c4b-b7a0-69e655bf31f5-openshift-service-ca\") pod \"maas-api-79c474dfdd-p6w4n\" (UID: \"d8038bec-aa91-4c4b-b7a0-69e655bf31f5\") " pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" Sep 11 19:28:24.040624 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:24.040594 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8038bec-aa91-4c4b-b7a0-69e655bf31f5-metrics-certs\") pod \"maas-api-79c474dfdd-p6w4n\" (UID: \"d8038bec-aa91-4c4b-b7a0-69e655bf31f5\") " pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" Sep 11 19:28:24.040624 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:24.040615 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8038bec-aa91-4c4b-b7a0-69e655bf31f5-maas-api-tls\") pod \"maas-api-79c474dfdd-p6w4n\" (UID: \"d8038bec-aa91-4c4b-b7a0-69e655bf31f5\") " pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" Sep 11 19:28:24.047211 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:24.047184 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mfds\" (UniqueName: \"kubernetes.io/projected/d8038bec-aa91-4c4b-b7a0-69e655bf31f5-kube-api-access-2mfds\") pod \"maas-api-79c474dfdd-p6w4n\" (UID: \"d8038bec-aa91-4c4b-b7a0-69e655bf31f5\") " pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" Sep 11 19:28:24.172976 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:24.172897 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" Sep 11 19:28:24.444273 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:24.444250 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n"] Sep 11 19:28:24.446031 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:28:24.445990 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8038bec_aa91_4c4b_b7a0_69e655bf31f5.slice/crio-64a93ee0e47c16fb76375cb2c6e06f910f877037cf48122cd43a9d1988762ae2 WatchSource:0}: Error finding container 64a93ee0e47c16fb76375cb2c6e06f910f877037cf48122cd43a9d1988762ae2: Status 404 returned error can't find the container with id 64a93ee0e47c16fb76375cb2c6e06f910f877037cf48122cd43a9d1988762ae2 Sep 11 19:28:24.620891 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:24.620854 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" event={"ID":"d8038bec-aa91-4c4b-b7a0-69e655bf31f5","Type":"ContainerStarted","Data":"64a93ee0e47c16fb76375cb2c6e06f910f877037cf48122cd43a9d1988762ae2"} Sep 11 19:28:25.626986 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:25.626944 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" event={"ID":"d8038bec-aa91-4c4b-b7a0-69e655bf31f5","Type":"ContainerStarted","Data":"4b441378dade712a0ce774dc37fe7348a6dfd08d0b944d3878ef304a20c96560"} Sep 11 19:28:25.627402 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:25.627015 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" Sep 11 19:28:25.643839 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:25.643788 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" podStartSLOduration=2.2608618480000002 podStartE2EDuration="2.64377119s" totalImagesPullingTime="382.909342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:28:25.643110648 +0000 UTC m=+2411.096916614" watchObservedRunningTime="2026-09-11 19:28:25.64377119 +0000 UTC m=+2411.097577146" Sep 11 19:28:31.637244 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:31.637208 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79c474dfdd-p6w4n" Sep 11 19:28:31.685784 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:31.685751 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z"] Sep 11 19:28:31.686015 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:31.685990 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" podUID="1328a435-9c84-4fc7-9140-bdf4f5ee41d2" containerName="maas-api" containerID="cri-o://5cc3551b7d1fdfb343aa463f5d6c21ff39090ff8c70966af352dd0ef75c982bd" gracePeriod=30 Sep 11 19:28:31.926368 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:31.926345 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:32.015162 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.015131 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-maas-api-tls\") pod \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\" (UID: \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\") " Sep 11 19:28:32.015353 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.015185 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkvgw\" (UniqueName: \"kubernetes.io/projected/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-kube-api-access-hkvgw\") pod \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\" (UID: \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\") " Sep 11 19:28:32.015353 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.015211 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-metrics-certs\") pod \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\" (UID: \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\") " Sep 11 19:28:32.015353 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.015303 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-openshift-service-ca\") pod \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\" (UID: \"1328a435-9c84-4fc7-9140-bdf4f5ee41d2\") " Sep 11 19:28:32.015727 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.015701 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1328a435-9c84-4fc7-9140-bdf4f5ee41d2" (UID: "1328a435-9c84-4fc7-9140-bdf4f5ee41d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:28:32.017306 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.017279 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1328a435-9c84-4fc7-9140-bdf4f5ee41d2" (UID: "1328a435-9c84-4fc7-9140-bdf4f5ee41d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:28:32.017415 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.017358 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1328a435-9c84-4fc7-9140-bdf4f5ee41d2" (UID: "1328a435-9c84-4fc7-9140-bdf4f5ee41d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:28:32.017415 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.017372 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-kube-api-access-hkvgw" (OuterVolumeSpecName: "kube-api-access-hkvgw") pod "1328a435-9c84-4fc7-9140-bdf4f5ee41d2" (UID: "1328a435-9c84-4fc7-9140-bdf4f5ee41d2"). InnerVolumeSpecName "kube-api-access-hkvgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:28:32.116861 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.116825 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-openshift-service-ca\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:28:32.116861 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.116854 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-maas-api-tls\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:28:32.116861 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.116865 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkvgw\" (UniqueName: \"kubernetes.io/projected/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-kube-api-access-hkvgw\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:28:32.117112 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.116874 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1328a435-9c84-4fc7-9140-bdf4f5ee41d2-metrics-certs\") on node \"ip-10-0-143-253.ec2.internal\" DevicePath \"\"" Sep 11 19:28:32.664909 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.664878 2556 generic.go:356] "Generic (PLEG): container finished" podID="1328a435-9c84-4fc7-9140-bdf4f5ee41d2" containerID="5cc3551b7d1fdfb343aa463f5d6c21ff39090ff8c70966af352dd0ef75c982bd" exitCode=0 Sep 11 19:28:32.665349 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.664927 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" Sep 11 19:28:32.665349 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.664961 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" event={"ID":"1328a435-9c84-4fc7-9140-bdf4f5ee41d2","Type":"ContainerDied","Data":"5cc3551b7d1fdfb343aa463f5d6c21ff39090ff8c70966af352dd0ef75c982bd"} Sep 11 19:28:32.665349 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.665020 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z" event={"ID":"1328a435-9c84-4fc7-9140-bdf4f5ee41d2","Type":"ContainerDied","Data":"c3f67d601117f3424a6e5305e151eede4e3f3f7fe0226c5a84be482a78d0a283"} Sep 11 19:28:32.665349 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.665048 2556 scope.go:117] "RemoveContainer" containerID="5cc3551b7d1fdfb343aa463f5d6c21ff39090ff8c70966af352dd0ef75c982bd" Sep 11 19:28:32.675578 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.675552 2556 scope.go:117] "RemoveContainer" containerID="5cc3551b7d1fdfb343aa463f5d6c21ff39090ff8c70966af352dd0ef75c982bd" Sep 11 19:28:32.675886 ip-10-0-143-253 kubenswrapper[2556]: E0911 19:28:32.675865 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc3551b7d1fdfb343aa463f5d6c21ff39090ff8c70966af352dd0ef75c982bd\": container with ID starting with 5cc3551b7d1fdfb343aa463f5d6c21ff39090ff8c70966af352dd0ef75c982bd not found: ID does not exist" containerID="5cc3551b7d1fdfb343aa463f5d6c21ff39090ff8c70966af352dd0ef75c982bd" Sep 11 19:28:32.675970 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.675900 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc3551b7d1fdfb343aa463f5d6c21ff39090ff8c70966af352dd0ef75c982bd"} err="failed to get container status \"5cc3551b7d1fdfb343aa463f5d6c21ff39090ff8c70966af352dd0ef75c982bd\": rpc error: code = NotFound desc = could not find container \"5cc3551b7d1fdfb343aa463f5d6c21ff39090ff8c70966af352dd0ef75c982bd\": container with ID starting with 5cc3551b7d1fdfb343aa463f5d6c21ff39090ff8c70966af352dd0ef75c982bd not found: ID does not exist" Sep 11 19:28:32.686268 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.686235 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z"] Sep 11 19:28:32.691922 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:32.691898 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-79856dc5dd-zqz6z"] Sep 11 19:28:33.171710 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:33.171668 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1328a435-9c84-4fc7-9140-bdf4f5ee41d2" path="/var/lib/kubelet/pods/1328a435-9c84-4fc7-9140-bdf4f5ee41d2/volumes" Sep 11 19:28:51.547749 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:51.547719 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c9988978f-8s8t5_94668d09-5d8d-4a84-a81e-6d6571619c58/copy-manifests/0.log" Sep 11 19:28:51.554053 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:51.554027 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c9988978f-8s8t5_94668d09-5d8d-4a84-a81e-6d6571619c58/manager/0.log" Sep 11 19:28:52.128267 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:52.128234 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79c474dfdd-p6w4n_d8038bec-aa91-4c4b-b7a0-69e655bf31f5/maas-api/0.log" Sep 11 19:28:52.342524 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:52.342497 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819220-bpgbc_5e694c7b-4b7c-45fa-9ca9-eca594a7cb5e/cleanup/0.log" Sep 11 19:28:52.549268 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:52.549245 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-qckvx_1f4f6a7e-46be-42f3-bd1c-adc4e72b3060/postgres/0.log" Sep 11 19:28:53.757915 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:53.757879 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-fjcfw_c8650c92-d57f-4877-b214-b58cefdc073c/manager/0.log" Sep 11 19:28:54.429795 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:54.429758 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6m92t_53506b1c-b55c-4c79-9cd7-f672455e3c08/discovery/0.log" Sep 11 19:28:54.868391 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:54.868353 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-k9fdk_929571ec-dc93-4566-b194-c2b97d95a535/payload-pre-processing/0.log" Sep 11 19:28:55.649901 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:55.649865 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-mvqzs_c54d6285-3940-44e1-ac75-876677a401cb/main/0.log" Sep 11 19:28:55.759836 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:28:55.759808 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-ppckr_9eedf2a6-9683-46f6-afbc-464ac0e312bc/main/0.log" Sep 11 19:29:01.081225 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:01.081190 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:29:01.147288 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:01.147255 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" limit=1024 Sep 11 19:29:03.916852 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:03.916819 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r2xrg_953eaca9-ab99-4bce-b95a-ab3d3fbad183/global-pull-secret-syncer/0.log" Sep 11 19:29:04.006928 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:04.006901 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4gnvd_ad502b23-3af5-4907-8f22-1c3c619555ca/konnectivity-agent/0.log" Sep 11 19:29:04.138179 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:04.138148 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-253.ec2.internal_bdd962729f012a9148b1a889b407d7c0/haproxy/0.log" Sep 11 19:29:09.079874 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:09.079842 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-fjcfw_c8650c92-d57f-4877-b214-b58cefdc073c/manager/0.log" Sep 11 19:29:10.840221 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:10.840130 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-xqr2r_6c357c65-fe42-4459-b6e3-f80fbe5ac120/kube-state-metrics/0.log" Sep 11 19:29:10.859614 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:10.859576 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-xqr2r_6c357c65-fe42-4459-b6e3-f80fbe5ac120/kube-rbac-proxy-main/0.log" Sep 11 19:29:10.882769 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:10.882745 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-xqr2r_6c357c65-fe42-4459-b6e3-f80fbe5ac120/kube-rbac-proxy-self/0.log" Sep 11 19:29:10.919367 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:10.919345 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-57dc7f4fdd-72fr6_639dfeb3-c288-4575-89a8-80647f6437d6/metrics-server/0.log" Sep 11 19:29:10.965802 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:10.965779 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-hqr5j_ce6c5f05-f277-4907-a227-8296e0ab8f4a/monitoring-plugin/0.log" Sep 11 19:29:11.007007 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.006980 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l6cw9_a5be5812-3f19-4d44-adf4-3c5de1c36cf1/node-exporter/0.log" Sep 11 19:29:11.034670 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.034645 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l6cw9_a5be5812-3f19-4d44-adf4-3c5de1c36cf1/kube-rbac-proxy/0.log" Sep 11 19:29:11.056883 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.056866 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l6cw9_a5be5812-3f19-4d44-adf4-3c5de1c36cf1/init-textfile/0.log" Sep 11 19:29:11.403434 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.403357 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f/prometheus/0.log" Sep 11 19:29:11.426684 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.426661 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f/config-reloader/0.log" Sep 11 19:29:11.450434 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.450414 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f/thanos-sidecar/0.log" Sep 11 19:29:11.477002 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.476982 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f/kube-rbac-proxy-web/0.log" Sep 11 19:29:11.499935 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.499914 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f/kube-rbac-proxy/0.log" Sep 11 19:29:11.522155 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.522126 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f/kube-rbac-proxy-thanos/0.log" Sep 11 19:29:11.544029 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.544014 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3aec0263-5fdd-4cfc-a6cf-fa07dbcaca9f/init-config-reloader/0.log" Sep 11 19:29:11.631625 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.631604 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-rwl8w_035c9e13-4e2a-41dc-b036-d50ec80a0cb6/prometheus-operator-admission-webhook/0.log" Sep 11 19:29:11.736311 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.736258 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b95dc5876-cchsf_d9743bf3-35ce-43c1-8fcd-331b9f01538c/thanos-query/0.log" Sep 11 19:29:11.756988 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.756967 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b95dc5876-cchsf_d9743bf3-35ce-43c1-8fcd-331b9f01538c/kube-rbac-proxy-web/0.log" Sep 11 19:29:11.777817 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.777796 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b95dc5876-cchsf_d9743bf3-35ce-43c1-8fcd-331b9f01538c/kube-rbac-proxy/0.log" Sep 11 19:29:11.799287 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.799268 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b95dc5876-cchsf_d9743bf3-35ce-43c1-8fcd-331b9f01538c/prom-label-proxy/0.log" Sep 11 19:29:11.819994 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.819978 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b95dc5876-cchsf_d9743bf3-35ce-43c1-8fcd-331b9f01538c/kube-rbac-proxy-rules/0.log" Sep 11 19:29:11.842837 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:11.842819 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b95dc5876-cchsf_d9743bf3-35ce-43c1-8fcd-331b9f01538c/kube-rbac-proxy-metrics/0.log" Sep 11 19:29:13.814904 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:13.814871 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m"] Sep 11 19:29:13.815329 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:13.815315 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1328a435-9c84-4fc7-9140-bdf4f5ee41d2" containerName="maas-api" Sep 11 19:29:13.815373 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:13.815331 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1328a435-9c84-4fc7-9140-bdf4f5ee41d2" containerName="maas-api" Sep 11 19:29:13.815433 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:13.815424 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1328a435-9c84-4fc7-9140-bdf4f5ee41d2" containerName="maas-api" Sep 11 19:29:13.818801 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:13.818785 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:13.821099 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:13.821081 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6fmxz\"/\"kube-root-ca.crt\"" Sep 11 19:29:13.821655 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:13.821640 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6fmxz\"/\"openshift-service-ca.crt\"" Sep 11 19:29:13.821748 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:13.821655 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6fmxz\"/\"default-dockercfg-bb7j4\"" Sep 11 19:29:13.834240 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:13.833907 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m"] Sep 11 19:29:13.987824 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:13.987785 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38e54047-c38e-425f-9519-ceda367d510f-podres\") pod \"perf-node-gather-daemonset-cr87m\" (UID: \"38e54047-c38e-425f-9519-ceda367d510f\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:13.987985 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:13.987853 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38e54047-c38e-425f-9519-ceda367d510f-proc\") pod \"perf-node-gather-daemonset-cr87m\" (UID: \"38e54047-c38e-425f-9519-ceda367d510f\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:13.987985 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:13.987884 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38e54047-c38e-425f-9519-ceda367d510f-sys\") pod \"perf-node-gather-daemonset-cr87m\" (UID: \"38e54047-c38e-425f-9519-ceda367d510f\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:13.987985 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:13.987908 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhmwz\" (UniqueName: \"kubernetes.io/projected/38e54047-c38e-425f-9519-ceda367d510f-kube-api-access-lhmwz\") pod \"perf-node-gather-daemonset-cr87m\" (UID: \"38e54047-c38e-425f-9519-ceda367d510f\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:13.987985 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:13.987926 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38e54047-c38e-425f-9519-ceda367d510f-lib-modules\") pod \"perf-node-gather-daemonset-cr87m\" (UID: \"38e54047-c38e-425f-9519-ceda367d510f\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:14.088968 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:14.088867 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38e54047-c38e-425f-9519-ceda367d510f-proc\") pod \"perf-node-gather-daemonset-cr87m\" (UID: \"38e54047-c38e-425f-9519-ceda367d510f\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:14.088968 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:14.088905 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38e54047-c38e-425f-9519-ceda367d510f-sys\") pod \"perf-node-gather-daemonset-cr87m\" (UID: \"38e54047-c38e-425f-9519-ceda367d510f\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:14.088968 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:14.088925 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhmwz\" (UniqueName: \"kubernetes.io/projected/38e54047-c38e-425f-9519-ceda367d510f-kube-api-access-lhmwz\") pod \"perf-node-gather-daemonset-cr87m\" (UID: \"38e54047-c38e-425f-9519-ceda367d510f\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:14.089270 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:14.089014 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38e54047-c38e-425f-9519-ceda367d510f-sys\") pod \"perf-node-gather-daemonset-cr87m\" (UID: \"38e54047-c38e-425f-9519-ceda367d510f\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:14.089270 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:14.089049 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38e54047-c38e-425f-9519-ceda367d510f-proc\") pod \"perf-node-gather-daemonset-cr87m\" (UID: \"38e54047-c38e-425f-9519-ceda367d510f\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:14.089270 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:14.089048 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38e54047-c38e-425f-9519-ceda367d510f-lib-modules\") pod \"perf-node-gather-daemonset-cr87m\" (UID: \"38e54047-c38e-425f-9519-ceda367d510f\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:14.089270 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:14.089131 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38e54047-c38e-425f-9519-ceda367d510f-podres\") pod \"perf-node-gather-daemonset-cr87m\" (UID: \"38e54047-c38e-425f-9519-ceda367d510f\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:14.089270 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:14.089165 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38e54047-c38e-425f-9519-ceda367d510f-lib-modules\") pod \"perf-node-gather-daemonset-cr87m\" (UID: \"38e54047-c38e-425f-9519-ceda367d510f\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:14.089270 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:14.089228 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38e54047-c38e-425f-9519-ceda367d510f-podres\") pod \"perf-node-gather-daemonset-cr87m\" (UID: \"38e54047-c38e-425f-9519-ceda367d510f\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:14.096144 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:14.096120 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhmwz\" (UniqueName: \"kubernetes.io/projected/38e54047-c38e-425f-9519-ceda367d510f-kube-api-access-lhmwz\") pod \"perf-node-gather-daemonset-cr87m\" (UID: \"38e54047-c38e-425f-9519-ceda367d510f\") " pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:14.131630 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:14.131607 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:14.402527 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:14.402503 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m"] Sep 11 19:29:14.405199 ip-10-0-143-253 kubenswrapper[2556]: W0911 19:29:14.405171 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod38e54047_c38e_425f_9519_ceda367d510f.slice/crio-e6ee0ce1ba5a2fb49a4642c550d1244ef39a00f3c9568dfd5048ee3b0a367b2c WatchSource:0}: Error finding container e6ee0ce1ba5a2fb49a4642c550d1244ef39a00f3c9568dfd5048ee3b0a367b2c: Status 404 returned error can't find the container with id e6ee0ce1ba5a2fb49a4642c550d1244ef39a00f3c9568dfd5048ee3b0a367b2c Sep 11 19:29:14.864627 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:14.864591 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" event={"ID":"38e54047-c38e-425f-9519-ceda367d510f","Type":"ContainerStarted","Data":"d3addfa9e5582a0d8e74eb117e8b70c202e6a918d65b97e48852b7047f606f26"} Sep 11 19:29:14.865010 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:14.864642 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" event={"ID":"38e54047-c38e-425f-9519-ceda367d510f","Type":"ContainerStarted","Data":"e6ee0ce1ba5a2fb49a4642c550d1244ef39a00f3c9568dfd5048ee3b0a367b2c"} Sep 11 19:29:14.865010 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:14.864686 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:14.880605 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:14.880557 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" podStartSLOduration=1.8805283739999998 podStartE2EDuration="1.880528374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:14.878829178 +0000 UTC m=+2460.332635124" watchObservedRunningTime="2026-09-11 19:29:14.880528374 +0000 UTC m=+2460.334334328" Sep 11 19:29:15.610701 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:15.610670 2556 scope.go:117] "RemoveContainer" containerID="d97de469576f6b5d982ec6f0db6b48bf9e285f4fecf66d511ba548eaa2d87e07" Sep 11 19:29:16.222963 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:16.222933 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h928z_bebc8327-abaa-4039-96f8-90928ef2e27d/dns/0.log" Sep 11 19:29:16.243246 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:16.243217 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h928z_bebc8327-abaa-4039-96f8-90928ef2e27d/kube-rbac-proxy/0.log" Sep 11 19:29:16.318906 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:16.318879 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n8gcj_16a827b3-ee19-4675-ab27-8b47231c2ebc/dns-node-resolver/0.log" Sep 11 19:29:16.959769 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:16.959743 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vm7pc_908b068d-8d94-4fcb-8490-eb8b1ac59a90/node-ca/0.log" Sep 11 19:29:18.429144 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:18.429107 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6m92t_53506b1c-b55c-4c79-9cd7-f672455e3c08/discovery/0.log" Sep 11 19:29:18.540940 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:18.540912 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-k9fdk_929571ec-dc93-4566-b194-c2b97d95a535/payload-pre-processing/0.log" Sep 11 19:29:19.529392 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:19.529363 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g7ldv_ad0303b1-5952-4ffc-b1d1-d256b97ba1e6/serve-healthcheck-canary/0.log" Sep 11 19:29:20.041677 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:20.041646 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-kxkhf_9462a6cd-3ef7-4962-aaea-01017510998d/insights-operator/0.log" Sep 11 19:29:20.042221 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:20.042199 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-kxkhf_9462a6cd-3ef7-4962-aaea-01017510998d/insights-operator/1.log" Sep 11 19:29:20.208233 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:20.208209 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nvjvz_f3f67d93-49c4-45a9-803b-806c6446432c/kube-rbac-proxy/0.log" Sep 11 19:29:20.230709 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:20.230682 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nvjvz_f3f67d93-49c4-45a9-803b-806c6446432c/exporter/0.log" Sep 11 19:29:20.252250 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:20.252227 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nvjvz_f3f67d93-49c4-45a9-803b-806c6446432c/extractor/0.log" Sep 11 19:29:20.879320 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:20.879297 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6fmxz/perf-node-gather-daemonset-cr87m" Sep 11 19:29:22.924279 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:22.924247 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c9988978f-8s8t5_94668d09-5d8d-4a84-a81e-6d6571619c58/manager/0.log" Sep 11 19:29:22.947076 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:22.947054 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c9988978f-8s8t5_94668d09-5d8d-4a84-a81e-6d6571619c58/copy-manifests/0.log" Sep 11 19:29:24.405080 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:24.405043 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-744bc9d7dd-wbcxf_ba3b8052-b4d5-42e7-a609-dd8ed95b8d28/manager/0.log" Sep 11 19:29:24.773685 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:24.773604 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6m92t_53506b1c-b55c-4c79-9cd7-f672455e3c08/discovery/0.log" Sep 11 19:29:24.867464 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:24.867432 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-k9fdk_929571ec-dc93-4566-b194-c2b97d95a535/payload-pre-processing/0.log" Sep 11 19:29:26.911931 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:26.911900 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6m92t_53506b1c-b55c-4c79-9cd7-f672455e3c08/discovery/0.log" Sep 11 19:29:26.999300 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:26.999272 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-k9fdk_929571ec-dc93-4566-b194-c2b97d95a535/payload-pre-processing/0.log" Sep 11 19:29:29.635952 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:29.635925 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-tpxz7_d3d4ec18-eacd-43c7-a4fd-31e078bcc6bf/migrator/0.log" Sep 11 19:29:29.662183 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:29.662156 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-tpxz7_d3d4ec18-eacd-43c7-a4fd-31e078bcc6bf/graceful-termination/0.log" Sep 11 19:29:30.095564 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:30.095508 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-m2xrk_8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1/kube-storage-version-migrator-operator/1.log" Sep 11 19:29:30.096399 ip-10-0-143-253 kubenswrapper[2556]: I0911 19:29:30.096382 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-m2xrk_8f7101b1-479d-4ee5-9d3b-4df6a63dc9d1/kube-storage-version-migrator-operator/0.log"